Sieges tend to be decisive in Clan Battles. As you saw, a clan defending a siege scores points in three of the seven categories (holding an enemy PoP being out of reach for most).
When your opponents activated a siege the game sent a series of marches to attack their stronghold. Your opponents put troops into their stronghold and battles took place in which the attacking Jotunn marches killed some of your opponents troops and their troops killed some of the attacking troops. Your opponents scored points (and gained soul shards) for each attacker killed.
That you and your clanmate took no part in the battles does not matter.
To activate a siege a clan needs to have built a Tower of Odin in their stronghold. Once built the Tower can be developed through a total of (currently) six levels. At level one the number of Jotunn marches sent and the number and strength of the troops in the marches is lower and by level six the marches are more numerous with many more and stronger troops included.
As matters stand, when you see the draw for each CvC look at the figure on the top right of the opponents' clan's details. If the figure is 4 or above you will encounter a siege and can know in advance that your chance to win is minimal.
What you need to do is to develop a strategy. One approach at your present level is for you and your clanmate to reduce your clan's influence as low as you can. This will match you with less well developed opponents and your chance that the opponents have not yet advanced their SH to level 4 and built a Tower of Odin is better.
Alternatively you might prioritize developing your SH to level 4, build a Tower of Odin and advance it. This will only make sense if you have strong enough troops to defend well against the Jotunn marches. You will certainly need a well developed hero, a set of legendary Ubba gear, a well developed Raine the Wanderer and legendary gear for him.
It turns out that there is something of a tightrope to walk. You want to have a Tower which generates at least as many Jotunn marches as your opponents - but developing the Tower of Odin faster than your clan's ability to fight well in the siege is a mistake.
Keeping influence down and being cautious about each Tower of Odin advance is the way to go.
All this has an additional significance. Defending sieges is by far the easiest way to acquire soul shards and soul shards turn out to be critical for all development in the late middle game. It will be some time before you reach that point but developing a good strategy now will pay dividends later.