Music journalists have noted the theme of abuse and misogyny on Caligula: Writing for The Guardian, Ben Beaumont-Thomas noted that Hayter "[emphasizes] the rage and despair of survivors of abuse," while Jessie Jeffrey Dunn Robinelli of Tiny Mix Tapes wrote that the album "lives in the aftermath of intimate partner abuse, in the shivering convulsions of rage and fear it leaves, in the furious movement between identification with and rejection of the abuse and the abuser, in failures of all sorts." In an interview with Kerrang!, she stated that on the album she is "taking you through a cycle of abuse", and added that she was looking at "abusive power, madness, depravity, narcissism" present in politics and communities, as well as "[herself] and [her] my own kind of madness as a result of trauma."
Themes of "hate, vengeance and violence" are also prominently explored. Hans Kim of PopMatters considered that the album "is not exactly about healing trauma, but rather, it tries to better voice the enduring effects of it", explaining that Hayter "unapologetically expresses her lasting anger throughout the album". Robinelli described it as "'kill your rapist' not as self-care, not as political slogan, but as the emotional and physical manifestation of pain beyond self", citing the line "I'm going to throw your body in the fucking river", while Will Gotsegen of Spin opined that "Hayter treats revenge as an imperative, and the death of her abusers as an inevitability." The track "Fragrant Is My Many Flower’d Crown" contains lyrics from Billy Bragg's "Tender Comrade", which the abuser that Caligula is written about named an album of his after. On the track "If the Poison Won't Take You My Dogs Will", Hayter reworked the Jonestown death tape as she "wanted to replicate also the really fucked up ambient sound in the death tape, the people screaming, the panicked energy", as well as "the way Jim Jones kept manipulating until the very end" in the song's lyrics, which examine suicide.
Musically, the album combines elements of classical music and extreme music genres, with Pitchfork editor Jenn Pelly describing it as "a murderous amalgam of opera, metal, and noise". Robinelli considered it "constructed from the most melodramatic sides of doom metal, noise, and neo-classical traditions" and noted an assistance of bands such as the Body, Full of Hell, and Uniform on the album. Similarly, Spyros Stasis of PopMatters highlighted her use of noise, power electronics, and dark ambient with her "choral background and neo-classical influences [...] to construct the maze-like anatomy of her work", while Noisey's Leah Mandel further noted the elements of black metal and Baroque music on the album. Hayter stated in an interview that she "knew this record had to go all the way, sonically and otherwise", and added that she "wanted to create something that sounded very large and powerful, consuming and overwhelming." She has also contrasted it to her previous album, revealing that she "limited [the] electronic and industrial" style found on All Bitches Die, and "added more stringency and acoustic instruments".
Hayter's vocal performance has been highlighted and praised by music critics in their reviews: discussing the track "Do You Doubt Me Traitor", James MacKinnon of Kerrang! called it "by far Kristin's most captivating weapon", describing how "she stacks vocal harmonies [...] before disintegrating into a throat-shredding wail that channels every moment of pain, injustice and powerlessness into a sonic bullet", while Pelly dubbed it "the sound of trauma". Beaumont-Thomas noted how the track "Sorrow! Sorrow! Sorrow!" "sees her split her voice apart like a throat singer, while other sections recall the subtle ululations of traditional Gaelic song". Hayter explained in an interview with Louder that "many many times on Caligula we kept the ugliest take", recalling how she used a recording of her voice for "Faithful Servant Friend of Christ" specifically …