Simulating the human heart's electromechanics using finite-element methods requires several hours per beat in a supercomputer. A graph neural network (GNN) model trained on simulated data can approximate the cardiac passive mechanics and speed up these calculations. To achieve robustness to varying geometries and benefit from this problem's physical symmetries, we propose an SE(3) equivariant GNN capable of accelerating simulations. To make the representation of the input data invariant, we represent the fibre orientation vectors as scalars. We use ventricular coordinates to define robust invariant local bases, namely, we use the apex-to-base and transmural vectors, and their cross-product. Then, we project the fibre vectors into these local bases and obtain the invariant embeddings. Other node-wise properties in the input data for this problem were already invariant. Our GNN model takes as input the resting biventricular geometry as a graph G and its node-wise properties to predict the node-wise displacements produced by the passive mechanics of the diastolic filling phase. We tested the training capacity and prediction cost of our GNN on one geometry under various conditions. Our GNN predicted the passive deformations in ~5 seconds on a desktop machine, resulting in a speed-up of 3 orders of magnitude compared to the finite-element methods solver used for generating the training data. However, we observed displacement prediction errors with high correlations to regions with different material and contractile properties (e.g., the valvular plugs) and regions with changing boundary conditions (the base to valves region). Our results illustrate the advantages and limitations of GNNs for accelerating passive cardiac mechanics simulations with homogeneous tissue properties.