I am extremely excited to write this post: it demonstrates how to use a technology that I strongly believe is key in protecting users and keeping centralized services accountable, and it’s the result of years of effort by me, the TrustFabric team at Google, the Sigsum team at Glasklar, and many others.

This article by Geomys is being cross-posted on Cryptography Dispatches.

Let’s start by defining the goal: we want a secure and convenient way to fetch age public keys for other people and services.

The easiest and most usable way to achieve that is to build a centralized keyserver: a web service where you log in with your email address to set your public key, and other people can look up public keys by email address.

Trusting the third party that operates the keyserver lets you solve identity, authentication, and spam by just delegating the responsibilities of checking email ownership and implementing rate limiting. The keyserver can send a link to the email address, and whoever receives it is authorized to manage the public key(s) bound to that address.

I had Claude Code build the base service, because it’s simple and not the interesting part of what we are doing today. There’s nothing special in the implementation: just a Go server, an SQLite database, a lookup API, a set API protected by a CAPTCHA that sends an email authentication link, and a Go CLI that calls the lookup API.

Transparency logs and accountability for centralized services

A lot of problems are shaped like this and are much more solvable with a trusted third party: PKIs, package registries, voting systems… Sometimes the trusted third party is encapsulated behind a level of indirection, and we talk about Certificate Authorities, but it’s the same concept.

Centralization is so appealing that even the OpenPGP ecosystem embraced it: after the SKS pool was killed by spam, a new OpenPGP keyserver was built which is just a centralized, email-authenticated database of public keys. Its FAQ claims they don’t wish to be a CA, but also explains they don’t support the (dubiously effective) Web-of-Trust at all, so effectively they can only act as a trusted third party.

The obvious downside of a trusted third party is, well, trust. You need to trust the operator, but also whoever will control the operator in the future, and also the operator’s security practices. That’s asking a lot, especially these days, and a malicious or compromised keyserver could provide fake public keys to targeted victims with little-to-no chance of detection.

Transparency logs are a technology for applying cryptographic accountability to centralized systems with no UX degradation.

A transparency log or tlog is an append-only, globally consistent list of entries, with efficient cryptographic proofs of inclusion and consistency. The log operator appends entries to the log, which can be tuples like (package, version, hash) or (email, public key). The clients verify an inclusion proof before accepting an entry, guaranteeing that the log operator will have to stand by that entry in perpetuity and to the whole world, with no way to hide it or disown it. As long as someone who can check the authenticity of the entry will eventually check (or “monitor”) the log, the client can trust that malfeasance will be caught.

Effectively, a tlog lets the log operator stake their reputation to borrow time for collective, potentially manual verification of the log’s entries. This is a middle-ground between impractical local verification mechanisms like the Web of Trust, and fully trusted mechanisms like centralized X.509 PKIs.

If you’d like a longer introduction, my Real World Crypto 2024 talk presents both the technical functioning and abstraction of modern transparency logs.

There is a whole ecosystem of interoperable tlog tools and publicly available infrastructure built around C2SP specifications. That’s what we are going to use today to add a tlog to our keyserver.

If you want to catch up with the tlog ecosystem, my 2025 Transparency.dev Summit Keynote maps out the tools, applications, and specifications.

tlogs vs Certificate Transparency vs Key Transparency

If you are familiar with Certificate Transparency, tlogs are derived from CT, but with a few major differences. Most importantly, there is no separate entry producer (in CT, the CAs) and log operator; moreover, clients check actual inclusion proofs instead of SCTs; finally, there are stronger split-view protections, as we will see below. The Static CT API and Sunlight CT log implementation were a first successful step in moving CT towards the tlog ecosystem, and a proposed design called Merkle Tree Certificates redesigns the WebPKI to have tlog-like and tlog-interoperable transparency.

In my experience, it’s best not to think about CT when learning about tlogs. A better production example of a tlog is the Go Checksum Database, where Google logs the module name, version, and hash for every module version observed by the Go Modules Proxy. The module fetches happen over regular HTTPS, so there is no publicly-verifiable proof of their authenticity. Instead, the central party appends every observation to the tlog, so that any misbehavior can be caught. The go get command verifies inclusion proofs for every module it downloads, protecting 100% of the ecosystem, without requiring module authors to manage keys.

Katie Hockman gave a great talk on the Go Checksum Database at GopherCon 2019.

You might also have heard of Key Transparency. KT is an overlapping technology that was deployed by Apple, WhatsApp, and Signal amongst others. It has similar goals, but picks different tradeoffs that involve significantly more complexity, in exchange for better privacy and scalability in some settings.

A tlog for our keyserver

Ok, so how do we apply a tlog to our email-based keyserver?

It’s pretty simple, and we can do it with a 250-line diff using Tessera and Torchwood. Tessera is a general-purpose tlog implementation library, which can be backed by object storage or a POSIX filesystem. For our keyserver, we’ll use the latter backend, which stores the whole tlog in a directory according to the c2sp.org/tlog-tiles specification.

s, err := note.NewSigner(os.Getenv("LOG_KEY"))
if err != nil {
    log.Fatalln("failed to create checkpoint signer:", err)
}
v, err := torchwood.NewVerifierFromSigner(os.Getenv("LOG_KEY"))
if err != nil {
    log.Fatalln("failed to create checkpoint verifier:", err)
}
policy := torchwood.ThresholdPolicy(2, torchwood.OriginPolicy(v.Name()),
    torchwood.SingleVerifierPolicy(v))

driver, err := posix.New(ctx, posix.Config{
    Path: *logPath,
})
if err != nil {
    log.Fatalln("failed to create log storage driver:", err)
}

// Since this is a low-traffic but interactive server, disable batching to
// remove integration latency for the first request. Keep a 1s checkpoint
// interval not to hit the witnesses too often; this will be observed only
// if two requests come in quick succession. Finally, only publish a
// checkpoint once a day if there are no new entries, making the average qps
// on witnesses low. Poll for new checkpoints quickly since it should be
// just a read from a hot filesystem cache.
checkpointInterval := 1 * time.Second
if testing.Testing() {
    checkpointInterval = 100 * time.Millisecond
}
appender, shutdown, logReader, err := tessera.NewAppender(ctx, driver, tessera.NewAppendOptions().
    WithCheckpointSigner(s).
    WithBatching(1, tessera.DefaultBatchMaxAge).
    WithCheckpointInterval(checkpointInterval).
    WithCheckpointRepublishInterval(24*time.Hour))
if err != nil {
    log.Fatalln("failed to create log appender:", err)
}
defer shutdown(context.Background())
awaiter := tessera.NewPublicationAwaiter(ctx, logReader.ReadCheckpoint, 25*time.Millisecond)

Every time a user sets their key, we append an encoded (email, public key) entry to the tlog, and we store the tlog entry index in the database.

+    // Add to transparency log
+    if strings.ContainsAny(email, "\n") {
+        http.Error(w, "Invalid email format", http.StatusBadRequest)
+        return
+    }
+    entry := tessera.NewEntry(fmt.Appendf(nil, "%s\n%s\n", email, pubkey))
+    index, _, err := s.awaiter.Await(r.Context(), s.appender.Add(r.Context(), entry))
+    if err != nil {
+        http.Error(w, "Failed to add to transparency log", http.StatusInternalServerError)
+        log.Printf("transparency log error: %v", err)
+        return
+    }
+
     // Store in database
-    if err := s.storeKey(email, pubkey); err != nil {
+    if err := s.storeKey(email, pubkey, int64(index.Index)); err != nil {
         http.Error(w, "Failed to store key", http.StatusInternalServerError)
         log.Printf("database error: %v", err)
         return
     }

The lookup API produces a proof from the index and provides it to the client.

func (s *Server) makeSpicySignature(ctx context.Context, index int64) ([]byte, error) {
    checkpoint, err := s.reader.ReadCheckpoint(ctx)
    if err != nil {
        return nil, fmt.Errorf("failed to read checkpoint: %v", err)
    }
    c, _, err := torchwood.VerifyCheckpoint(checkpoint, s.policy)
    if err != nil {
        return nil, fmt.Errorf("failed to parse checkpoint: %v", err)
    }
    p, err := tlog.ProveRecord(c.N, index, torchwood.TileHashReaderWithContext(
        ctx, c.Tree, tesserax.NewTileReader(s.reader)))
    if err != nil {
        return nil, fmt.Errorf("failed to create proof: %v", err)
    }
    return torchwood.FormatProof(index, p, checkpoint), nil
}

The proof follows the c2sp.org/tlog-proof specification. It looks like this

c2sp.org/tlog-proof@v1
index 1
CJdjppwZSa2A60oEpcdj/OFjVQyrkP3fu/Ot2r6smg0=

keyserver.geomys.org
2
HtFreYGe2VBtaf3Vf0AG0DAwEZ+H92HQqrx4dkrzk0U=

— keyserver.geomys.org FrMVCWmHnYfHReztLams2F3HUY6UMub3c5xu7+e8R8SAk9cxPKAB1fsQ6gFM16xwkvZ8p5aWaBf8km+M20eHErSfGwI=

and it combines a checkpoint (a signed snapshot of the log at a certain size), the index of the entry in the log, and a proof of inclusion of the entry in the checkpoint.

The client CLI receives the proof from the lookup API, checks the signature on the checkpoint from the built-in log public key, hashes the expected entry, and checks the inclusion proof for that hash and checkpoint. It can do all this without interacting further with the log.

vkey := os.Getenv("AGE_KEYSERVER_PUBKEY")
if vkey == "" {
    vkey = defaultKeyserverPubkey
}
v, err := note.NewVerifier(vkey)
if err != nil {
    fmt.Fprintf(os.Stderr, "Error: invalid keyserver public key: %v\n", err)
    os.Exit(1)
}
policy := torchwood.ThresholdPolicy(2, torchwood.OriginPolicy(v.Name()),
    torchwood.SingleVerifierPolicy(v))

// Verify spicy signature
entry := fmt.Appendf(nil, "%s\n%s\n", result.Email, result.Pubkey)
if err := torchwood.VerifyProof(policy, tlog.RecordHash(entry), []byte(result.Proof)); err != nil {
    return "", fmt.Errorf("failed to verify key proof: %w", err)
}

If you squint, you can see that the proof is really a “fat signature” for the entry, which you verify with the log’s public key, just like you’d verify an Ed25519 or RSA signature for a message. I like to call them spicy signatures to stress how tlogs can be deployed anywhere you can deploy regular digital signatures.

Monitoring

What’s the point of all this though? The point is that anyone can look through the log to make sure the keyserver is not serving unauthorized keys for their email address! Indeed, just like backups are useless without restores and signatures are useless without verification, tlogs are useless without monitoring. That means we need to build tooling to monitor the log.

On the server side, it takes two lines of code, to expose the Tessera POSIX log directory.

// Serve tlog-tiles log
fs := http.StripPrefix("/tlog/", http.FileServer(http.Dir(*logPath)))
mux.Handle("GET /tlog/", fs)

On the client side, we add an -all flag to the CLI that reads all matching entries in the log.

func monitorLog(serverURL string, policy torchwood.Policy, email string) ([]string, error) {
    f, err := torchwood.NewTileFetcher(serverURL+"/tlog", torchwood.WithUserAgent("age-keylookup/1.0"))
    if err != nil {
        return nil, fmt.Errorf("failed to create tile fetcher: %w", err)
    }
    c, err := torchwood.NewClient(f)
    if err != nil {
        return nil, fmt.Errorf("failed to create torchwood client: %w", err)
    }

    // Fetch and verify checkpoint
    signedCheckpoint, err := f.ReadEndpoint(context.Background(), "checkpoint")
    if err != nil {
        return nil, fmt.Errorf("failed to read checkpoint: %w", err)
    }
    checkpoint, _, err := torchwood.VerifyCheckpoint(signedCheckpoint, policy)
    if err != nil {
        return nil, fmt.Errorf("failed to parse checkpoint: %w", err)
    }

    // Fetch all entries up to the checkpoint size
    var pubkeys []string
    for i, entry := range c.AllEntries(context.Background(), checkpoint.Tree, 0) {
        e, rest, ok := strings.Cut(string(entry), "\n")
        if !ok {
            return nil, fmt.Errorf("malformed log entry %d: %q", i, string(entry))
        }
        k, rest, ok := strings.Cut(rest, "\n")
        if !ok || rest != "" {
            return nil, fmt.Errorf("malformed log entry %d: %q", i, string(entry))
        }
        if e == email {
            pubkeys = append(pubkeys, k)
        }
    }
    if c.Err() != nil {
        return nil, fmt.Errorf("error fetching log entries: %w", c.Err())
    }

    return pubkeys, nil
}

To enable effective monitoring, we also normalize email addresses by trimming spaces and lowercasing them, since users are unlikely to monitor all the variations. We do it before sending the login link, so normalization can't lead to impersonation.

// Normalize email
email = strings.TrimSpace(strings.ToLower(email))

A complete monitoring story would involve 3rd party services that monitor the log for you and email you if new keys are added, like gopherwatch and Source Spotter do for the Go Checksum Database, but the -all flag is a start.

The full change involves 5 files changed, 251 insertions(+), 6 deletions(-), plus tests, and includes a new keygen helper binary, the required database schema and help text and API changes, and web UI changes to show the proof.

Privacy with VRFs

We created a problem by implementing this tlog, though: now all the email addresses of our users are public! While this is ok for module names in the Go Checksum Database, allowing email address enumeration in our keyserver is a non-starter for privacy and spam reasons.

We could hash the email addresses, but that would still allow offline brute-force attacks. The right tool for the job is a Verifiable Random Function. You can think of a VRF as a hash with a private and public key: only you can produce a hash value, using the private key, but anyone can check that it’s the correct (and unique) hash value, using the public key.

Overall, implementing VRFs takes less than 130 lines using the c2sp.org/vrf-r255 instantiation based on ristretto255, implemented by filippo.io/mostly-harmless/vrf-r255 (pending a more permanent location). Instead of the email address, we include the VRF hash in the log entry, and we save the VRF proof in the database.

+        // Compute VRF hash and proof
+        vrfProof := s.vrf.Prove([]byte(email))
+        vrfHash := base64.StdEncoding.EncodeToString(vrfProof.Hash())
+
         // Add to transparency log
-        entry := tessera.NewEntry(fmt.Appendf(nil, "%s\n%s\n", email, pubkey))
+        entry := tessera.NewEntry(fmt.Appendf(nil, "%s\n%s\n", vrfHash, pubkey))
         index, _, err := s.awaiter.Await(r.Context(), s.appender.Add(r.Context(), entry))
         if err != nil {
             http.Error(w, "Failed to add to transparency log", http.StatusInternalServerError)
         }

         // [...]

         // Store in database
-        if err := s.storeKey(email, pubkey, int64(index.Index)); err != nil {
+        if err := s.storeKey(email, pubkey, int64(index.Index), vrfProof.Bytes()); err != nil {
             http.Error(w, "Failed to store key", http.StatusInternalServerError)
             log.Printf("database error: %v", err)
             return
         }

The tlog proof format has space for application-specific opaque extra data, so we can store the VRF proof there, to keep the tlog proof self-contained.

-    return torchwood.FormatProof(index, p, checkpoint), nil
+    return torchwood.FormatProofWithExtraData(index, vrfProof, p, checkpoint), nil

In the client CLI, we extract the VRF hash from the tlog proof’s extra data and verify it’s the correct hash for the email address.

+    // Compute and verify VRF hash
+    vrfProofBytes, err := torchwood.ProofExtraData([]byte(result.Proof))
+    if err != nil {
+        return "", fmt.Errorf("failed to extract VRF proof: %w", err)
+    }
+    vrfProof, err := vrf.NewProof(vrfProofBytes)
+    if err != nil {
+        return "", fmt.Errorf("failed to parse VRF proof: %w", err)
+    }
+    vrfHash, err := vrfKey.Verify(vrfProof, []byte(email))
+    if err != nil {
+        return "", fmt.Errorf("failed to verify VRF proof: %w", err)
+    }
+
     // Verify spicy signature
-    entry := fmt.Appendf(nil, "%s\n%s\n", result.Email, result.Pubkey)
+    vrfHashB64 := base64.StdEncoding.EncodeToString(vrfHash)
+    entry := fmt.Appendf(nil, "%s\n%s\n", vrfHashB64, result.Pubkey)
     if err := torchwood.VerifyProof(policy, tlog.RecordHash(entry), []byte(result.Proof)); err != nil {
         return "", fmt.Errorf("failed to verify key proof: %w", err)
     }

How do we do monitoring now, though? We need to add a new API that provides the VRF hash (and proof) for an email address.

     mux.HandleFunc("GET /manage", srv.handleManage)
     mux.HandleFunc("POST /setkey", srv.handleSetKey)
     mux.HandleFunc("GET /api/lookup", srv.handleLookup)
+    mux.HandleFunc("GET /api/monitor", srv.handleMonitor)
     mux.HandleFunc("POST /api/verify-token", srv.handleVerifyToken)

func (s *Server) handleMonitor(w http.ResponseWriter, r *http.Request) {
    email := r.URL.Query().Get("email")
    if email == "" {
        http.Error(w, "Email parameter required", http.StatusBadRequest)
        return
    }

    // Return as JSON
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(map[string]any{
        "email":     email,
        "vrf_proof": s.vrf.Prove([]byte(email)).Bytes(),
    })
}

On the client side, we use that API to obtain the VRF proof, we verify it, and we look for the VRF hash in the log instead of looking for the email address.

Attackers can still enumerate email addresses by hitting the public lookup or monitor API, but they’ve always been able to do that: serving such a public API is the point of the keyserver! With VRFs, we restored the original status quo: enumeration requires brute-forcing the online, rate-limited API, instead of having a full list of email addresses in the tlog (or hashes that can be brute-forced offline).

VRFs have a further benefit: if a user requests to be deleted from the service, we can’t remove their entries from the tlog, but we can stop serving the VRF for their email address from the lookup and monitor APIs. This makes it impossible to obtain the key history for that user, or even to check if they ever used the keyserver, but doesn’t impact monitoring for other users.

The full change adding VRFs involves 3 files changed, 125 insertions(+), 13 deletions(-), plus tests.

Anti-poisoning

We have one last marginal risk to mitigate: since we can’t ever remove entries from the tlog, what if someone inserts some unsavory message in the log by smuggling it in as a public key, like age1llllllllllllllrustevangellsmstrlkef0rcellllllllllllq574n08?

Protecting against this risk is called anti-poisoning. The risk to our log is relatively small, public keys have to be Bech32-encoded and short, so an attacker can’t usefully embed images or malware. Still, it’s easy enough to neutralize it: instead of the public keys, we put their hashes in the tlog entry, keeping the original public keys in a new table in the database, and serving them as part of the monitor API.

         // Compute VRF hash and proof
         vrfProof := s.vrf.Prove([]byte(email))
-        vrfHash := base64.StdEncoding.EncodeToString(vrfProof.Hash())
+
+        // Keep track of the unhashed key
+        if err := s.storeHistory(email, pubkey); err != nil {
+            http.Error(w, "Failed to store key history", http.StatusInternalServerError)
+            log.Printf("database error: %v", err)
+            return
+        }

         // Add to transparency log
-        entry := tessera.NewEntry(fmt.Appendf(nil, "%s\n%s\n", vrfHash, pubkey))
+        h := sha256.New()
+        h.Write([]byte(pubkey))
+        entry := tessera.NewEntry(h.Sum(vrfProof.Hash())) // vrf-r255(email) || SHA-256(pubkey)
         index, _, err := s.awaiter.Await(r.Context(), s.appender.Add(r.Context(), entry))

It’s very important that we persist the original key in the database before adding the entry to the tlog. Losing the original key would be indistinguishable from refusing to provide a malicious key to monitors.

On the client side, to do a lookup we just hash the public key when verifying the inclusion proof. To monitor in -all mode, we match the hashes against the list of original public keys provided by the server through the monitor API.

     var result struct {
         Email    string   `json:"email"`
         VRFProof []byte   `json:"vrf_proof"`
+        History  []string `json:"history"`
     }

+    // Prepare map of hashes of historical keys
+    historyHashes := make(map[[32]byte]string)
+    for _, pk := range result.History {
+        h := sha256.Sum256([]byte(pk))
+        historyHashes[h] = pk
+    }

     // Fetch all entries up to the checkpoint size
     var pubkeys []string
     for i, entry := range c.AllEntries(context.Background(), checkpoint.Tree, 0) {
-        e, rest, ok := strings.Cut(string(entry), "\n")
-        if !ok {
-            return nil, fmt.Errorf("malformed log entry %d: %q", i, string(entry))
-        }
-        k, rest, ok := strings.Cut(rest, "\n")
-        if !ok || rest != "" {
-            return nil, fmt.Errorf("malformed log entry %d: %q", i, string(entry))
-        }
-        if e == base64.StdEncoding.EncodeToString(vrfHash) {
-            pubkeys = append(pubkeys, k)
-        }
+        if len(entry) != 64+32 {
+            return nil, fmt.Errorf("invalid entry size at index %d", i)
+        }
+        if !bytes.Equal(entry[:64], vrfHash) {
+            continue
+        }
+        pk, ok := historyHashes[([32]byte)(entry[64:])]
+        if !ok {
+            return nil, fmt.Errorf("found unknown public key hash in log at index %d", i)
+        }
+        pubkeys = append(pubkeys, pk)
     }

Our final log entry format is vrf-r255(email) || SHA-256(pubkey). Designing the tlog entry is the most important part of deploying a tlog: it needs to include enough information to let monitors isolate all the entries relevant to them, but not enough information to pose privacy or poisoning threats.

The full change providing anti-poisoning involves 2 files changed, 93 insertions(+), 19 deletions(-), plus tests.

Non-equivocation and the Witness Network

We’re almost done! There’s still one thing to fix, and it used to be the hardest part.

To get the delayed, collective verification we need, all clients and monitors must see consistent views of the same log, where the log maintains its append-only property. This is called non-equivocation, or split-view protection. In other words, how do we stop the log operator from showing an inclusion proof for log A to a client, and then a different log B to the monitors?

Just like logging without a monitoring story is like signing without verification, logging without a non-equivocation story is just a complicated signature algorithm with no strong transparency properties.

This is the hard part because in the general case you can’t do it alone. Instead, the tlog ecosystem has the concept of witness cosigners: third-party operated services which cosign a checkpoint to attest that it is consistent with all the other checkpoints the witness observed for that log. Clients check these witness cosignatures to get assurance that—unless a quorum of witnesses is colluding with the log—they are not being presented a split-view of the log.

These witnesses are extremely efficient to operate: the log provides the O(log N) consistency proof when requesting a cosignature, and the witness only needs to store the O(1) latest checkpoint it observed. All the potentially intensive verification is deferred and delegated to monitors, which can be sure to have the same view as all clients thanks to the witness cosignatures.

This efficiency makes it possible to operate witnesses for free as public benefit infrastructure. The Witness Network collects public witnesses and maintains an open list of tlogs that the witnesses automatically configure.

For the Geomys instance of the keyserver, I generated a tlog key and then I sent a PR to the Witness Network to add the following lines to the testing log list.

vkey keyserver.geomys.org+16b31509+ARLJ+pmTj78HzTeBj04V+LVfB+GFAQyrg54CRIju7Nn8
qpd 1440
contact keyserver-tlog@geomys.org

This got my log configured in a handful of witnesses, from which I picked three to build the default keyserver witness policy.

log keyserver.geomys.org+16b31509+ARLJ+pmTj78HzTeBj04V+LVfB+GFAQyrg54CRIju7Nn8
witness TrustFabric transparency.dev/DEV:witness-little-garden+d8042a87+BCtusOxINQNUTN5Oj8HObRkh2yHf/MwYaGX4CPdiVEPM https://api.transparency.dev/dev/witness/little-garden/
witness Mullvad witness.stagemole.eu+67f7aea0+BEqSG3yu9YrmcM3BHvQYTxwFj3uSWakQepafafpUqklv https://witness.stagemole.eu/
witness Geomys witness.navigli.sunlight.geomys.org+a3e00fe2+BNy/co4C1Hn1p+INwJrfUlgz7W55dSZReusH/GhUhJ/G https://witness.navigli.sunlight.geomys.org/
group public 2 TrustFabric Mullvad Geomys
quorum public

The policy format is based on Sigsum’s policies, and it encodes the log’s public key and the witnesses’ public keys (for the clients) and submission URLs (for the log).

Tessera supports these policies directly. When minting a new checkpoint, it will reach out in parallel to all the witnesses, and return the checkpoint once it satisfies the policy. Configuration is trivial, and the added latency is minimal (less than one second).

+    witnessPolicy := defaultWitnessPolicy
+    if path := os.Getenv("LOG_WITNESS_POLICY"); path != "" {
+        witnessPolicy, err = os.ReadFile(path)
+        if err != nil {
+            log.Fatalln("failed to read witness policy file:", err)
+        }
+    }
+    witnesses, err := tessera.NewWitnessGroupFromPolicy(witnessPolicy)
+    if err != nil {
+        log.Fatalln("failed to create witness group from policy:", err)
+    }

     // [...]

     appender, shutdown, logReader, err := tessera.NewAppender(ctx, driver, tessera.NewAppendOptions().
         WithCheckpointSigner(s).
         WithBatching(1, tessera.DefaultBatchMaxAge).
         WithCheckpointInterval(checkpointInterval).
-        WithCheckpointRepublishInterval(24*time.Hour))
+        WithCheckpointRepublishInterval(24*time.Hour).
+        WithWitnesses(witnesses, nil))

On the client side, we can use Torchwood to parse the policy and use it directly with VerifyProof in place of the policy we were manually constructing from the log’s public key.

-    vkey := os.Getenv("AGE_KEYSERVER_PUBKEY")
-    if vkey == "" {
-        vkey = defaultKeyserverPubkey
-    }
+    policyBytes := defaultPolicy
+    if policyPath := os.Getenv("AGE_KEYSERVER_POLICY"); policyPath != "" {
+        p, err := os.ReadFile(policyPath)
+        if err != nil {
+            fmt.Fprintf(os.Stderr, "Error: failed to read policy file: %v\n", err)
+            os.Exit(1)
+        }
+        policyBytes = p
+    }
-    v, err := note.NewVerifier(vkey)
-    if err != nil {
-        fmt.Fprintf(os.Stderr, "Error: invalid keyserver public key: %v\n", err)
-        os.Exit(1)
-    }
-    policy := torchwood.ThresholdPolicy(2, torchwood.OriginPolicy(v.Name()), torchwood.SingleVerifierPolicy(v))
+    policy, err := torchwood.ParsePolicy(policyBytes)
+    if err != nil {
+        fmt.Fprintf(os.Stderr, "Error: invalid policy: %v\n", err)
+        os.Exit(1)
+    }

Again, if you squint you can see that just like tlog proofs are spicy signatures, the policy is a spicy public key. Verification is a deterministic, offline function that takes a policy/public key and a proof/signature, just like digital signature verification!

The policies are a DAG that can get complex to match even the strictest uptime requirements. For example, you can require 3 out of 10 witness operators to cosign a checkpoint, where each operator can use any 1 out of N witness instances to do so.

The full change implementing witnessing involves 5 files changed, 43 insertions(+), 11 deletions(-), plus tests.

Summing up

We started with a simple centralized email-authenticated keyserver, and we turned it into a transparent, privacy-preserving, anti-poisoning, and witness-cosigned service.

We did that in four small steps using Tessera, Torchwood, and various C2SP specifications.

cmd/age-keyserver: add transparency log of stored keys
    5 files changed, 259 insertions(+), 8 deletions(-)
cmd/age-keyserver: use VRFs to hide emails in the log
    3 files changed, 125 insertions(+), 13 deletions(-)
cmd/age-keyserver: hash age public key to prevent log poisoning
    2 files changed, 93 insertions(+), 19 deletions(-)
cmd/age-keyserver: add witness cosigning to prevent split-views
    5 files changed, 43 insertions(+), 11 deletions(-)

Overall, it took less than 500 lines.

7 files changed, 472 insertions(+), 9 deletions(-)

The UX is completely unchanged: there are no keys for users to manage, and the web UI and CLI work exactly like they did before. The only difference is the new -all functionality of the CLI, which allows holding the log operator accountable for all the public keys it could ever have presented for an email address.

The result is deployed live at keyserver.geomys.org.

Future work: efficient monitoring and revocation

This tlog system still has two limitations:

1. To monitor the log, the monitor needs to download it all. This is probably fine for our little keyserver, and even for the Go Checksum Database, but it’s a scaling problem for the Certificate Transparency / Merkle Tree Certificates ecosystem.

2. The inclusion proof guarantees that the public key is in the log, not that it’s the latest entry in the log for that email address. Similarly, the Go Checksum Database can’t efficiently prove the Go Modules Proxy /list response is complete.

We are working on a design called Verifiable Indexes which plugs on top of a tlog to provide verifiable indexes or even map-reduce operations over the log entries. We expect VI to be production-ready before the end of 2026, while everything above is ready today.

Even without VI, the tlog provides strong accountability for our keyserver, enabling a secure UX that would have simply not been possible without transparency.

I hope this step-by-step demo will help you apply tlogs to your own systems. If you need help, you can join the Transparency.dev Slack.

Keynote: The tlog Ecosystem

The event kicked off with an insightful keynote delivered by Filippo Valsorda, outlining the current state of affairs in transparency log ecosystems. Filippo gave a thorough overview of everything the transparency community has now, covering specifications, code, shared infrastructure and existing transparency applications, before looking to the future and discussing what comes next. With his keynote, Filippo gave context to the work being done by everyone in the room, tied together much of the content that followed in the next couple of days, and explained how it all fits into the bigger tlog ecosystem picture.

Certificate Transparency pushing forward

Following the keynote, the first day of the summit focussed primarily on Certificate Transparency (CT). A wonderfully balanced set of talks included points of view from every type of actor that participates in the CT ecosystem, giving the audience a fully fleshed out picture of how CT functions today.

Log operators were represented by Philippe Boneff and Roger Ng discussing running TesseraCT - a static CT log implementation. Matthew McPherrin represented Let’s Encrypt as both a log operator and a Certificate Authority, talking about the challenges and specific insights that being both provides. Although not a CT log operator himself, Dennis Jackson provided a novel idea for how to reduce the load on log operators by serving the content of CT logs as static ct tiles over BitTorrent.

Next up, User Agents and CT enforcers included Mustafa Emre Acer presenting how Chrome decides which CT logs qualify to be in their log list, and Roger Ng discussing CT enforcement in Android 16.

Andrew Ayer, who has been keeping log operators honest for many years now, talked about patterns seen in the various CT log failures he has detected as a log monitor. Certificate monitoring was covered by Program Committee Chair Rasmus Dahlberg in his talk about filtering out the noise of regular legitimate certificate issuance notifications as certificate lifetimes decrease. Then in a fun twist on monitoring, Bhushan Lokhande talked about using CT log data for a different purpose - to inspect brand logos and trademarks. His talk demonstrated that CT logs expose previously unseen but potentially interesting datasets.

To complete the CT ecosystem, and after a last minute unexpected circumstance that meant that he was unable to attend the summit in person, Program Committee Chair Martin Hutchinson dialled in remotely to contribute his talk about Verifiable Indexes - the key piece still missing from the CT ecosystem. Once in place, the CT ecosystem will be what it was originally envisaged to be.

After talking to us about things going bad in the world of CT, Joe DeBlasio then encouraged us to look forwards by presenting Merkle Tree Certificates (MTCs) - an early internet draft coauthored by many of the big players in the existing CT community, to plan a redesign for how CT will function in a post-quantum future. Certificate transparency is the longest established and most stable transparency ecosystem there is, and as such, drastic design changes rarely happen. However, as we look forward to what will eventually become a post-quantum world, CT will need to adapt along with everything else.

Transparency applications left, right and centre

For me personally, the second and third days of the summit were extremely special. As a former participant in the CT ecosystem and community, I left the transparency community in 2022 when non-CT transparency applications were few and far between, and those that did exist were just beginning to feel their way through what shape a transparency ecosystem should take. Having returned to the community only a few months ago, hearing talks on such a breadth of transparency applications fully opened my eyes to how far the community and technology has come since then.

To begin, our host in Gothenburg, Fredrik Strömberg, talked about his work on hardware transparency for the Tilitis TKey and HSM, and gave a demo (everyone loves a demo!) Adit Sachde followed this to discuss how confidential computing can protect virtual machines in the cloud, and how this could benefit the all important witness network as it begins to scale, by opening up the option for witnesses to be run in the cloud.

In the Website Transparency portion of the day, Michael Rosenberg, Dennis Jackson and Giulio Berra discussed the existing issues with cryptography in web applications, and presented ideas for how to bring properties such as integrity, consistency and transparency to web-based experiences.

Our next topic was package management, with Mechiel Lukkien taking us through how gopherwatch monitors Go modules, Hayden Blauzvern discussing the software supply chain and transparency for package registries in general, and Holger Levsen summarising the state of reproducible builds, and inviting the community to collaborate on bringing transparency into associated projects.

After lunch on the second day of the summit, we had one of the more whimsical moments of the conference. A life-sized cardboard cutout of Martin Hutchinson (our Program Committee Chair who was unable to attend last minute) arrived, and, much to everyone’s delight, it was established that cardboard Martin would be joining us all for dinner that night.

After this brief comical interlude, Alexis Hancock discussed the very real impact transparency could have on accountability within Digital ID systems that are being created by governments all over the world.

This was followed by talks on Binary/Firmware Transparency, including an entertaining talk and demo by Andrea Barisani and Daniele Bianco on implementing transparency in UEFI boot managers, Tom Binder discussing binary transparency based on signed endorsements - something Google has tailored to their use case of managing the boundary between open-source and internal repositories at Google, Billy Lau bringing us up to speed on Android’s Firmware Binary Transparency, and an energetic talk from Tiziano Santoro on distributed Content Addressable Storage.

To complete the wider transparency topics set, on the final day of the summit Melissa Chase gave a wonderful overview of Key Transparency (KT), with Brendan McMillion following it up with why the KT protocol being standardised by the IETF helps remove the need for an independent and trustworthy third party to help operate the system, and Elena Pagnin presenting a novel idea for achieving split-view protection in centralised transparency logs.

My expertise on these topics is limited, so I have held back from going further in summarising the content of these talks for fear of misrepresenting them. However, I strongly encourage you to check out each and every talk on the transparency.dev youtube channel.

Building together

While all of the talk content at the summit was top notch, it was the moments in and around the talks that really captured the spirit of the community. People were discussing ideas, collaborations and future work during every single break, every social dinner, every possible moment available. They couldn’t get enough of it.

Over the three days, we held two slots for breakout sessions, during which attendees came together in groups to discuss whatever topics they felt would be most beneficial to discuss in person. Topics included witnessing, MTCs, CT, transparency for web applications, Tilitis hardware, KT, and reproducible builds. These turned out to be some of the most popular sessions of the whole event, with everyone grumbling when encouraged back to the main room to move on to the next session. In our community, people thrive when given the opportunity to work together.

In Filippo’s keynote, he emphasised that ‘the hard part is the part that you can’t build alone.’ He meant it in terms of software - for most systems you just build the thing and put it out there, but in transparency systems you have to have third parties involved for the system to work. But I think that with that quote he has also stumbled upon the foundation of what makes this community great. Everyone here understands and accepts that, for transparency to work, we can’t do it alone. And although the hard part may be the part that you can’t build alone, with an incredible community of brilliant individuals all pulling together, the hard part becomes a hell of a lot easier.

A huge thank you to the hosts of the Transparency.dev Summit 2025, Glasklar Teknik, and our sponsors, Google, Google Chrome, Chalmers University of Technology, the University of Gothenburg and Trail of Bits, without whom the summit could not have happened.

For over a decade our team has been building and running transparency logs. What began with purpose-built Certificate Transparency (CT) logs soon led to the generalisation and broader adoption of these principles by other ecosystems. To encourage and facilitate this adoption we provided a general purpose transparency log implementation, Trillian, which has underpinned ecosystems like CT, Go's SumDB, and Sigstore ever since.

The growth and success of these ecosystems has driven demand for transparency logs that support higher throughput while also being cheaper and easier to operate. Tessera is now ready to meet that demand.

Key Features

• Ease of use: As a library rather than a microservice architecture, Tessera is built directly into application code. This gives you the freedom to build and deploy your application in the way that makes most sense for you.

• Tiles API: The C2SP tlog-tiles API is central to Tessera, helping you to build transparency-enabled applications that automatically benefit from many of the lessons learned over the last 10 years, and can easily integrate with shared services such as witnessing.

• Scalable architecture: Our design philosophy enables Tessera to play to the strengths of the target environment. Writes take an idiomatic approach, and reads are highly cacheable and, in many cases, able to be served directly from the underlying infrastructure.

• Multi-environment support: Logs built with Tessera can run natively in Cloud (GCP or AWS), or on-prem with Vanilla S3+MySQL or POSIX filesystem backends.

• Resilience and Availability: Applications can run multiple instances without compromising log storage, resulting in improved service resilience and availability in the face of frontend issues.

• Customizable log personalities: You can build transparency applications tailored to your specific use-case.

• Integrated support for witnessing: C2SP-compliant witnessing is supported out of the box, allowing easy integration with either the growing public witness network, or private witnesses.

• Reliable support: Tessera v1.0 is released with the Google Open Source breaking changes policy, meaning reliable library maintenance and support.

Performance

Tessera has proven to be stable at a high-throughput of writes in both our own testing environment, as well as long-running instances of our Tessera-based Certificate Transparency log implementation, TesseraCT.

With write-throughputs in excess of 10,000 entries per second backed by POSIX, and 5,000 entries/s on GCP, Tessera scales to meet most deployment scenarios. More detailed notes are available in the repo, and in the article by IPng discussing their own Tessera-based deployment.

Getting started with Tessera

Tessera is the path we recommended for building new transparency logs. To get started, check out the Codelab for a simple step-by-step guide to setting up your first log, writing a few entries to it, and then inspecting its contents.

Here's what others have to say:

• "Tessera has provided a fantastic developer experience while building the next version of Sigstore's signature transparency log. We've been able to scale up easily and reduce complexity in our infrastructure."

  -- Sigstore TL

• "Tessera was a breeze to work with! The API is straightforward and the storage backends are easy to import and swap for development and production."
  -- Adit Sachde

• "IPng Networks in Switzerland has deployed the first self-hosted Tessera-based static CT log, and reported a successful deployment in the Certificate Transparency community. The underlying Tessera library is blazingly fast compared to alternatives, reaching easily 18K leaves/sec, while the global WebPKI throughput is two orders of magnitude smaller. TesseraCT, built on this library, scales very well, allowing for fine grained configuration and off the shelf tools like nginx to serve the read-path. More details in https://ipng.ch/s/articles/2025/08/24/certificate-transparency-part-3-operations/ ."
  -- IPng Networks

We’d love to hear from you!

If you have any questions or would like guidance on working with Tessera please reach out to the team by posting questions on the transparency.dev community slack. We’re here to help, and hope that Tessera 1.0 will be instrumental in improving the verifiability of your systems.

At this stage we are releasing the alpha of TesseraCT, ready for early testing and use by others.

What is TesseraCT?

TesseraCT is a CT server that implements the Static CT API. It is built on top of Tessera, our Go library for tile-based transparency logs.

Historically, operating CT logs has been complex and costly, but TesseraCT changes that by providing a modern CT implementation that incorporates lessons learned from operating real-world logs at scale.

Key features

• Tile-based architecture and API: Enables efficient reads and writes, following the Static CT API C2SP specification.

• Cloud-native & on-prem: Supports AWS, Google Cloud, POSIX filesystems, or vanilla S3/MySQL storage systems.

• High uptime: Engineered to offer high availability, as required by user agents. While TesseraCT can run as a single instance, it will also safely run with multiple concurrent instances for higher availability.

• Low operational overhead: Fewer moving parts and easier setup than Trillian+CTFE. At its simplest, TesseraCT requires only a single server rather than three, and is simpler to configure.

• Future-proof: Designed for the highest level of availability, and to scale easily as certificate lifetimes get shorter and the volume of certificates issued increases.

• Fast merging: Entries are integrated into the log within seconds, with the option to wait for submissions to be fully integrated before client requests return.

Where can it run?

TesseraCT can be run on Google Cloud Platform (GCP), Amazon Web Services (AWS), POSIX filesystems, or using vanilla S3+MySQL storage systems. We have tested each of these implementations using various deployment setups, and have confirmed that they are able to handle the current certificate submission rate seen by existing logs, of roughly 300 certificates per second. At the higher end, our staging GCP logs happily accept up to 1.7k QPS, and a local POSIX deployment was tested up to 10k QPS! More details and reproducible performance tests are available in the TesseraCT repo.

We recommend you choose which platform to use with TesseraCT based on your existing production environment:

Tile-Based Logs

Tile-based logs became popular in the CT world due to the caching benefits they provide, making read requests to the log simpler and cheaper to respond to. Even Trillian, used by our existing RFC 6962 CT log implementation, relies on a tiled architecture under the hood. However, TesseraCT directly exposes C2SP compliant tiles through the static CT API.

In contrast to the microservices-based approach of Trillian and the CTFE, TesseraCT offers a fresh architecture with a flexible, and simplified deployment model. This is a transformative step for CT log infrastructure, for existing and new log operators.

TesseraCT builds on the heritage of Trillian+CTFE, and was designed after much discussion with current CT log operators. It is just one of a growing number of CT log implementations that conform to the static CT API specification, including the trailblazing Sunlight, Azul, Itko and CompactLog. Having multiple log implementations provides diversity and resilience, and makes it possible to run static CT API logs on a variety of platforms, resulting in a healthier Certificate Transparency ecosystem overall.

Try out TesseraCT!

We’ve launched the alpha release of TesseraCT so that developers, log operators, and the CT ecosystem can test it as early as possible and provide feedback to be incorporated into future releases.

There are three TesseraCT staging logs available for testing:

• arche2025h1 at https://arche2025h1.staging.ct.transparency.dev

• arche2025h2 at https://arche2025h2.staging.ct.transparency.dev

• arche2026h1 at https://arche2026h1.staging.ct.transparency.dev

CT log operators: Try out the Arche logs, or run your own! See our Getting Started guide and architecture overview for guidance on how. Give TesseraCT a try and help us answer the most important question: does this reduce your pain?

CT monitors: The Arche logs are now available for monitoring. Let us know if you run into any problems querying the logs.

Certificate Authorities: Submit certificates to our Arche logs.

We’re excited to share this milestone and we welcome your feedback. TesseraCT is about making it less painful to operate transparency infrastructure, so try the alpha and tell us what hurts. Let’s make the next generation of transparency logs better, together.

Keep an eye on the TesseraCT repo to follow along, and reach out with any questions or feedback on the transparency.dev slack. We look forward to hearing from you!

What’s Next?

We are continuing to push TesseraCT's development towards the goal of production tile-based CT logs. Future versions of TesseraCT will include additional features such as witnessing and a root update process integrated with CCADB. Check out our roadmap for more information.

Tessera is a tile-based transparency log library designed to improve the experience of log-operators running logs in any transparency space. After announcing the alpha release of Trillian Tessera late last year, we are excited to present the latest development milestone - the beta release of Tessera - which takes us one step closer to that goal!

From Alpha to Beta: What’s Changed?

As with any beta release, the objective was to produce a version of Tessera ready for external users to test and provide feedback on, so that we can go on to produce a production-ready version of Tessera. With that in mind, the changes between the alpha and beta releases improve the usability of the API, the stability of the codebase, alongside adding a few new features and tools. Highlights include:

• Snappier name: We’ve dropped the ‘Trillian’ prefix, it’s now simply ‘Tessera’. Although inspired by lessons learned from running Trillian, Tessera is a project in its own right, now quite different from Trillian. It’s time for it to leave the Trillian nest and independently find its own way in the world. Plus, it’s shorter, and doesn't contain a hyphen, so less typing, and no aliasing required when importing the Go package.

• API Improvements: The Tessera API is more intuitive and streamlined, with types and functions that are unnecessary to expose to users of the codebase hidden away.

• Expanded anti-spam support: Anti-spam support for AWS and POSIX has been added alongside the previously existing GCP support.

• New tools: We’ve added an fsck tool for checking the self-consistency and integrity of tlog-tiles logs via HTTP. Additionally, an experimental mirroring tool has been introduced, initially supporting local filesystem destinations.

• Cleaner docs: Documentation improvements and updates ensure clarity and accuracy for users.

• Bug fixes: We’ve addressed a number of bugs, including fixes for missing errors, inconsistencies in function names, and issues with checkpoint publishing in GCP and AWS.

It’s Testing Time

The latest improvements to Tessera refine the developer experience to the point that it is now ready for beta-level testing. So far, we’ve validated the functionality of Tessera by successfully running a log on GCP, scaling to over 2 billion entries. In parallel, we’ve also run a log on AWS, growing it to 800 million entries, demonstrating reliable performance across both platforms.

We invite all existing log operators, along with anyone else with an interest in improving and supporting transparency ecosystems, to try spinning up a log using the beta version of Tessera, and let us know how it goes! Feedback on your experience is invaluable, as hearing from external users about how Tessera fares when used for real applications helps us ensure that the upcoming production release is stable and meets your needs and expectations.

How to Get Involved

Tessera is designed to make it easy to build and deploy new transparency logs on supported infrastructure. To get started, check out the Codelab, which provides a simple step-by-step guide to set up your first log, write a few entries to it, and then inspect its contents.

To start experimenting with Tessera you can choose one of these example personalities:

• POSIX: example of operating a log backed by a local filesystem. This example runs an HTTP web server that takes arbitrary data and adds it to a file-based log.

• MySQL: example of operating a log that uses MySQL. This example is most easily deployed via docker compose, allowing for easy setup and teardown

• GCP: example of operating a log running on GCP. This example can be deployed via terraform.

• AWS: example of operating a log running on AWS. This example can be deployed via terraform.

In addition, keep an eye on the Getting Started section in the Tessera README for future example personalities.

For any questions or guidance on working with Tessera please reach out to the team by posting questions on the transparency.dev community slack. We can’t wait to hear what you think!

Earlier this year we announced the second annual Transparency.dev Summit, to be hosted by Glasklar Teknik on 20th-22nd October 2025 in Gothenburg, Sweden. Led by Rasmus Dahlberg and Martin Hutchinson, this year’s program committee would like to welcome you to submit proposals for talks and presentations for the summit.

👉 Submit your proposal here

If you have a talk, presentation or idea you'd like to share at the summit, we'd love to hear it. Submissions are welcomed from first-time speakers, well established members of the transparency community, and everyone in between!

The official deadline is August 31st, but decisions will be being made and presentation proposals will be being accepted as they come in, so submit ideas sooner rather than later. We'd hate for the event to miss out on a brilliant topic because the schedule is already full!

We’re looking forward to seeing what you come up with!

Detecting witness private key misuse is a much harder problem than detecting private key misuse by log. Inclusion proofs, such as signed certificate timestamps, can be cross checked against the current state of the log, but witnesses do not store a history of the checkpoints they signed. To protect against private key material leaks, the armored witness project uses custom hardware.

Confidential computing makes it possible to provide similar guarantees for witnesses running in the cloud. The confidential witness project provides an implementation for running a witness on GCP with public audit proofs. A running witness can be found at buoyant-breeze.itko.dev with audit logs found in this repository.

What is confidential computing?

Confidential computing aims to provide a guarantee to a remote client that the server is running a specific version of software, relying on support from cloud hardware and software. In our case, GCP provides the Secure Space image.

This image uses AMD’s Secure Encrypted Virtualization technology to receive a signed attestation that it is isolated from the Compute Engine hypervisor and that it has not been tampered with. Next, it launches the container specified by the operator and injects an IAM token into the container with these attestations. The operator’s software can use this token with remote services to provide the remote service with details about the currently running version of software.

Confidential computing shifts the trust boundaries of traditional applications. Instead of simply assuming that the service operator is running a correct version of software, we can rely on attestations generated by AMD and GCP to know that the service operator is running the correct version of software. We still have to trust someone, but it is likely that we trust the operational practices of AMD and GCP more than we trust those of the witness operators.

Trust flow of a confidential witness

The confidential witness software runs within the secure space without any state. The private key material is generated by and stored in GCP’s Key Management Service, ensuring that the operator does not have a copy unbound by the access policies, and other state is stored in a Cloud Storage bucket, both of which are protected with a set of IAM rules.

  attribute_condition = 
    'STABLE' in assertion.submods.confidential_space.support_attributes &&
    assertion.swname=='CONFIDENTIAL_SPACE' &&
    assertion.submods.container.image_reference=='${var.shasum}' &&
    assertion.submods.gce.project_id=='${var.project_id}' &&
    assertion.submods.container.env.WITNESS_KEY=='${local.witness_key}' &&
    assertion.submods.container.env.WITNESS_NAME=='${local.witness_name}' &&
    assertion.submods.container.env.WITNESS_AUDIENCE=='${local.witness_audience}' &&
    '${google_service_account.witness_compute_engine.email}' in assertion.google_service_accounts

One of the attributes provided in the IAM token is the hash of the container image. This attribute allows us to strongly link the image to a specific Github Actions workflow and the specific hash of the upstream confidential witness repository using Cosign. We can now audit the source code that went into building the container, providing an opportunity to ensure that there is nothing in the source that will allow the witness to maliciously sign checkpoints.

On the other side, we also need to make sure that the IAM rules for the cloud storage bucket and KMS are set up correctly to only accept requests from a container running an image with a hash that is known to be good.

Here, we rely on a Github actions workflow that calls the GCP APIs to fetch audit logs and configuration information, and then publish the data to a git repository. By using Github Actions instead of relying on the operator to publish the information, we can ensure that the data has not been tampered with prior to publishing.

Putting it all together

Let’s walk through using these tools to verify the source of the Buoyant Breeze witness.

1. Look at the audit logs to find what version of software is allowed to access the private key for the witness stored in GCP’s Key Management Service.

ghcr.io/aditsachde/confidential-witness@sha256:b634433ac01a0f43c05bbeb257044b990fee51a3128a5a5310192a5bddc9bc2d

2. Use the github cli to verify the attestation. Here, we can also specify the source commit, which can be looked up at https://github.com/aditsachde/confidential-witness/attestations.

gh attestation verify oci://ghcr.io/aditsachde/confidential-witness@sha256:b634433ac01a0f43c05bbeb257044b990fee51a3128a5a5310192a5bddc9bc2d --repo aditsachde/confidential-witness --source-digest 5b89c70e59060b51763219e8315cd9c2b4c45f1c

3. Audit the source code to ensure that this container will only sign properly verified checkpoints.

4. Use the audit logs to verify that the operator has configured everything in the project correctly and to our expectations, so that there are no other principals authorized to access private key material.

Confidential Witness

The confidential witness project provides an implementation of this concept, adapting omniwitness to run in a GCP confidential space with publicly published audit logs. The implementation is still a prototype, and not yet ready for production, but shows how these tools can be applied to the witness ecosystem. Interested in running your own witness or getting involved in the project? Join the community on the transparency.dev Slack!

Postgres and ZFS

While many distributed databases boast large-scale capabilities, we chose PostgreSQL for its proven performance, our operational experience, and because alternatives offered no clear benefit for our needs. With today's hardware capabilities, we were confident that we could fit all certificate transparency data in a single shard for the foreseeable future, and possibly indefinitely. The database we currently operate contains approximately 20TB of storage and over 100 billion unique rows.

We run two replicas for the core database with AMD EPYC 9454Ps, 6NVMe drives and 1TB of memory in each replica.

We chose ZFS as our storage solution for PostgreSQL to leverage its compression, snapshots, performance characteristics, and send/receive capabilities. ZFS's default recordsize of 128K serves as an excellent baseline for maintaining optimal performance in terms of compression ratio, sequential scans, and system overhead.

However, PostgreSQL's default page size of 8K presented a challenge: we would experience significant read/write amplification. For each page operation, we'd need to read or write an entire 128K record, resulting in a 16x overhead for many operations.

Traditionally, database administrators are advised to reduce the ZFS recordsize to match or approximate PostgreSQL's page size. This approach, however, would result in very small ZFS records (8K or 16K), substantially increasing global overhead and nearly eliminating the compression benefits, one of the primary advantages that make ZFS attractive in the first place.

Instead, we took the opposite approach: increasing PostgreSQL's block size to get closer to ZFS's default recordsize. Postgres maximum blocksize is 32L, so we will use that.

This setup is not difficult to achieve and can be done using the "configure" script:

./configure --with-blocksize=32 [other-options]

Optimizing for Append-Only Operations

PostgreSQL's VACUUM process handles multiple critical "cleanup" operations, including recycling dead tuples, updating statistics, and preventing transaction ID wraparound.

We designed our schema to be as "append-only" as possible. This design choice significantly reduces the workload on VACUUM operations since fewer updates and deletes mean fewer dead tuples to clean up. We configured autovacuum to be very aggressive with the following settings:

- autovacuum_max_workers to the number of merklemap tables

- autovacuum_naptime 10s

- autovacuum_vacuum_cost_delay 1ms

- autovacuum_vacuum_cost_limit 2000

Handling Transaction ID Wraparound

Despite our aggressive autovacuum configuration, sometimes the system would still become I/O constrained due to the high rate of data ingestion. This constraint particularly affects transaction ID wraparound issues.

According to the PostgreSQL VACUUM-FOR-WRAPAROUND documentation: “…since transaction IDs have limited size (32 bits) a cluster that runs for a long time (more than 4 billion transactions) would suffer transaction ID wraparound: the XID counter wraps around to zero, and all of a sudden transactions that were in the past appear to be in the future — which means their output become invisible. In short, catastrophic data loss. (Actually the data is still there, but that's cold comfort if you cannot get at it.) To avoid this, it is necessary to vacuum every table in every database at least once every two billion transactions.”

We proactively monitor database age metrics continuously. When predetermined thresholds are crossed, we implement a dynamic throttling mechanism: reducing ingestion rate, prioritizing newer logs, and once autovacuum catches up we ramp up ingestion again. . Once stable, ingestion ramps up again. Our append-only schema design significantly helps this process by minimizing the workload for autovacuum operations.

Relation-less?

A frequent recommendation for scaling write operations in PostgreSQL is to use the COPY command, minimize relations and constraints, and essentially "just pipe the data in." While this advice is common, it doesn't represent the optimal approach for every use case.

Prioritizing Data Integrity

When developing Merklemap, one of our primary motivations for choosing PostgreSQL was to leverage its robust constraint enforcement and ACID properties. Consequently, we designed our schema as a well-normalized relational model rather than optimizing solely for insertion efficiency.

This design decision does necessitate significantly more read operations and database commits, but we consider this performance trade-off worthwhile. The benefits of data integrity, consistency, and the ability to enforce complex relationships outweigh the potential write performance gains of a denormalized approach.

Optimizing Commit Performance

To mitigate the impact of our high commit frequency, we utilize PostgreSQL's commit grouping capabilities through the combination of commit_delay and commit_siblings parameters. This configuration allows the database to batch multiple commits together, reducing overhead.

At Merklemap, we maintain commit_delay at 100ms and keep commit_siblings at its default value of 5. This setup effectively balances our need for timely transaction completion with the efficiency benefits of grouped commits.Handling Transaction ID Wraparound

Despite our aggressive autovacuum configuration, sometimes the system would still become I/O constrained due to the high rate of data ingestion. This constraint particularly affects transaction ID wraparound issues.

Avoiding Partitioning

While PostgreSQL supports table partitioning, we found its benefits too narrow to justify the complexity. Partitioning can ease deletions, but brings challenges in constraint handling, indexing, foreign key management, and query planning. For us, well-indexed regular tables with thoughtful query discipline proved more efficient and simpler to maintain.

Query Discipline at Scale

Our general queries execute extremely fast, typically returning results in just a few microseconds. This efficiency stems from our strict adherence to querying only indexed data. We've carefully designed our indexing strategy to support our most common access patterns while balancing the overhead of index size and insert performance impact.

With over 100+ billion rows, casual querying is a non-starter. We strictly query indexed data, use partial, covering and expression indexes, and maintain materialized views for common aggregations. We also enforce query governor policies to prevent expensive scans during peak usage periods.

Compression algorithm

We use lz4 for ZFS compression, as it proved to be computationally negligible for our workload while providing a compression ratio of approximately 2x. We evaluated zstd, but lz4 offered the optimal balance between performance impact and storage savings.

Resilient Ingestion Strategy

We follow a "fail-fast, retry-often" approach. While certificate transparency logs are generally reliable, like any remotely accessible service, they can experience downtime or performance issues. Fortunately, since certificates and pre-certificates are replicated across multiple logs, monitoring interruptions don't pose a significant problem.

Our operation resembles that of a traditional search engine, dynamically throttling our queries based on "perceived" load. When we detect that a log is slower than usual or returning errors, we adjust accordingly.

We implement intelligent retry mechanisms that activate when logs return to a healthy state, ensuring we fill any gaps in our data where we previously encountered errors. This approach handles scenarios where specific blocks in the logs consistently fail for a period while other blocks remain accessible.

Efficient Ingestion

We implement our ingest workers using Rust’s Tokio library. We carefully pool a minimal number of PostgreSQL connections to maintain optimal performance, following best practices outlined in EDB's connection pooling recommendations.

With this optimized approach, we only require 5-6 CPU cores to ingest 5-8 million entries per hour.

Historical Data Volume Explanation?

You might wonder why our database has accumulated such a large volume of data. While the current certificate transparency traffic averages around 1 million certificates per hour, we've been ingesting data available since the inception of certificate transparency.

Conclusion

At Merklemap, we believe that traditional relational databases like PostgreSQL can still be the right choice for specialized large-scale applications when properly configured and optimized. By housing over 100 billion rows of certificate transparency data in a single PostgreSQL shard, we think we’ve shown that thoughtful architecture decisions can outweigh the immediate appeal of distributed systems in specific use cases.

While our current infrastructure comfortably handles the present certificate transparency ecosystem, we continue to monitor growth trends and optimize our processes. The techniques outlined in this article aren't just applicable to certificate transparency, they represent valuable approaches for any PostgreSQL deployment dealing with high-volume workloads.

🗓 Save the Date: October 20–22, 2025

📍 Gothenburg, Sweden

🏠 Hosted by Glasklar Teknik

The inaugural Transparency.dev Summit held in October 2024 was a huge success - bringing together over 50 attendees to drive the future direction of transparency ecosystems. Hosted by Google’s TrustFabric team in London UK, the summit sparked insightful discussions and strong community momentum. Now, the hosting baton passes to another key community member: Glasklar Teknik, who will welcome us to Gothenburg, Sweden for the next chapter.

This year, we’re diving deeper into the infrastructure, operations, and future of transparency. Whether you’re running a transparency log, building new transparency ecosystems, or simply interested in learning more about how transparency builds trust for any digital domain, this summit is for you.

This year’s program committee will be led by Rasmus Dahlberg and Martin Hutchinson, helping shape the content and flow of the summit.

Certificate Transparency (CT) Day

Certificate Transparency is a cornerstone of the conference. The CT community is undergoing a major evolution towards tiled logs to enhance the robustness of the ecosystem. Transparency.dev summit will feature a dedicated CT day with emphasis on:

• Adopting static CT logs for transformative benefits to the ecosystem

• Learning about real world challenges faced by log operators and browsers

• Tools and monitor solutions that help digest CT logs

• Post-Quantum Readiness - how to evolve for a PQ world

Central Pillars of the Summit

In addition to CT, the summit will be structured around the three following pillars:

• Applications of Transparency - We're excited to have talks on existing and emerging use-cases:

  • Binary Transparency - This is a broad umbrella where we focus on 2 aspects:

    • Signature transparency logs such as Sigsum and Sigstore.

    • Package index logs such as GoSumDB.

  • Key Transparency - E.g., discussion of production deployments and open challenges.

  • Other uses - Please do not feel limited if your application's umbrella is not listed here.

• Core Transparency Technology - Verifiable data structures are at the core of all transparency applications. Implementations of verifiable logs, maps, and adjacent technologies such as witnesses, verifiers, and monitors are for example in scope.

• Transparency Future Roadmap - What’s next for transparency? We’ll explore the roadmap ahead for various transparency technologies and standards.

Who Should Attend?

This summit is for:

• People running or building transparency infrastructure

• Open source maintainers and security engineers

• Academic researchers

• Policy folks and ecosystem designers

• Anyone passionate about verifiability, auditability, and trust

Get Involved

We want to hear from you! If you’re interested in attending, speaking, or helping shape the agenda:

👉 Fill out the Interest Survey

Transparency.dev Summit 2025 is sponsored by Google and Mullvad VPN. We’re actively welcoming additional sponsors who share our mission to grow and support the Transparency.dev community. If your organization is interested in getting involved, please fill out our interest survey to request more information.

Stay tuned to our website: Transparency.dev Summit for updates on the call-for-papers (CFP) and registration. Also join us in the transparency-dev Slack to hear the latest on the summit.

Firefox is now enforcing Certificate Transparency on desktop platforms, taking a significant step towards a safer web. With this change, effective in version 135, Firefox will reject certificates that do not comply with CT requirements. This ensures that all certificates trusted by the browser meet high transparency standards.

What does this mean for Website Owners?

It ensures that any TLS certificate trusted by Firefox is logged and publicly discoverable in a Certificate Transparency log. If your website already follows best practices and uses CT-compliant certificates, you don’t need to take any additional action. However, if you’re unsure, here are some steps you can take:

1. Ensure your Certificate Authority (CA) supports CT logging - Most major CAs already comply, but if you are using an uncommon CA, verify their status.

2. Monitor your certificates - Use Certificate Transparency monitoring services and tools to ensure no unauthorized certificates which would be trusted by Firefox and other CT enforcing user agents are issued for your domain.

Firefox CT in Action

Certificate transparency information can be delivered either as:

• signed certificate timestamps (SCTs) embedded in the certificate itself, or

• SCTs stapled alongside the certificate (via the TLS handshake or in an OCSP response).

For a connection to succeed, sufficient certificate transparency information must be provided using either of these methods. See Firefox CT Policy for more details.

You can see CT enforcement in action for yourself using the ‘https://no-sct.badssl.com‘ test site which if accessed from Firefox v135 shows the error "MOZILLA_PKIX_ERROR_INSUFFICIENT_CERTIFICATE_TRANSPARENCY" to reflect the fact that the server does not send a Signed Certificate Timestamp (SCT) for the domain of the test site.

Image by Matthew McPherrin on Bluesky

Firefox Known CT Logs

Supporting CT in a browser requires verifying SCTs from an approved set of CT logs. This presents a non-trivial operational issue, especially as CT logs come and go over time. Each browser enforcing CT sets up their own user agent policy for log configuration. For CT in Firefox:

• The list of known trusted logs is derived from Chrome’s list and is automatically updated each week in prerelease versions of Firefox. This means CT log operators do not need to submit new logs to Firefox.

• To see what logs are in a particular version of Firefox, you can examine the history of Known CT Logs.

To learn more about Firefox CT policies, see Certificate Transparency in Firefox

Firefox & Tile-Based Logs

As the Certificate Transparency community moves toward tile-based logs - supporting static-ct-api logs in addition to RFC6962 logs - questions arise about whether Firefox will follow suit. On the Mozilla dev-security-policy mailing list, Dana Keeler provided a positive indication that Mozilla is open to this approach, especially as industry adoption increases. Keeler stated: “If it becomes clear that supporting static-ct-api logs is necessary to interoperate, we will probably allow them as well.”

Final Thoughts

The enforcement of Certificate Transparency in Firefox marks a major step forward in web security. With all major browsers now requiring certificates to be logged in CT logs, it makes it harder for attackers to abuse certificates without detection, and means a safer browsing experience for users. For website owners, it’s a reminder to stay vigilant and monitor CT logs for certificates covering their domains.

More broadly, Certificate Transparency continues to evolve and gain adoption across the industry. As browsers also start to look towards accepting tile-based logs, the CT ecosystem is becoming more robust, ensuring greater transparency and security for the web.

The CT ecosystem is evolving. The original specification, RFC6962, was published in June 2013. Subsequent projects such as Go’s SumDB extended the application of transparency logs to other places. The Static CT API takes the lessons learned from these projects and applies them back to CT, moving some expensive operations from server-side to client-side. This change drastically simplifies how CT logs function and makes them much cheaper to operate.

To better appreciate the impact of these changes, my CT log implementation, Itko, supports both RFC6962 and the Static CT API. Itko also has an operational instance which is available for querying and testing against.

Lessons learned

The Static CT API is a variant of the more generic tiled transparency log specification, a specification you can use to bring transparency to your own ecosystem – here are some of my takeaways which may be useful for you.

Community

First, the most important takeaway for me from this project is how welcoming and helpful the transparency community is. I would not have been able to build Itko without them. The Transparency.dev Slack and the CT policy mailing list are great places to learn, ask questions, and follow along on development.

Tooling

In 2024, the ecosystem was in a bit of a scattered state. The library and tools exist, but they are spread over a number of packages and projects. Itko uses components from Go’s SumDB, Trillian, and certificate-transparency-go.

Changing this is a focus for the ecosystem going into 2025, and large strides have already been made. Trillian Tessera is the focal point of this effort to make tiled transparency logs usable by all. Currently out in alpha, Trillian Tessera bundles all the common functionality needed by tiled transparency logs, allowing implementers to focus on the bits unique to their ecosystem.

Operational Simplicity

Tiled logs are operationally awesome. The only component in the read path is a filesystem, no database needed! And, static files means most requests hit the page cache, not disk. Itko ingests two million certificates and serves millions of requests per day on a quarter of a cpu core, because there is no database involved.

However, write path uptime is at odds with this simplicity. A transparency log needs all its entries to have a unique and ordered sequence number. This introduces a global counter that must be coordinated between nodes in a distributed system. There are a variety of approaches to tackle this problem, but with Itko, I instead chose to not tackle it.

The CT ecosystem baselines 99% uptime due to the presence of multiple redundant logs. Itko takes advantage of this by handling all write operations in a single process, with additional processes in standby mode ready to take over if a short period of unavailability is detected. Instead of targeting the highest uptime possible, a careful consideration of ecosystem requirements can allow logs to remove a lot of complexity and potential failure modes.

Unexpected visitors

Transparency logs are a great investigative research tool, which means quite a few folks monitor them. The Itko 2025 shard was meant to be a demo and is not considered to be a trusted log by Chrome or Safari. But surprisingly, the log was quickly serving more than 300gb of traffic a day as multiple people were monitoring it!

The number of requests your log receives will have an impact on the most economical choice of storage. Tiled logs fit well with object storage, with the benefit of never having to worry about downtime. However, object storage has a cost on every read and write request, which can be cost prohibitive for public good services like transparency logs. For Itko, I found a shared persistent disk to be a more economical choice, one that also resulted in better performance due to lower write latencies.

Testing

Having good test coverage is always useful, but having a comprehensive test suite is especially important when the servers and clients are written by different people.

When writing Itko, my first step was to write integration tests, adapting the test cases and the CT hammer tool from certificate-transparency-go. Then, I wrote the actual implementation, which allowed for a high degree of confidence in spec compliance. And it totally paid off. After the log passed the test cases, it worked with other OSS tools on the first try.

Just do it!

I had zero understanding of how transparency logs worked when I first started. Over the course of about 40 extremely scattered hours of work, I was able to get Itko to pass its spec compliance test cases. The trust guarantees that transparency logs provide make them seem a bit magical. But today, they are not very complicated to implement. Join the Slack, take a look at Trillian Tessera, and just dive in!

Resources:

• Itko Repository and Development Instance

• Join the Slack!

• Transparency.dev Roadmap

As we step into 2025, the rapid evolution of the digital landscape means security concerns continue to be of paramount importance. The Transparency.dev ecosystem has emerged as a pivotal community for fostering innovation in CT and beyond. This post takes a look at some key developments you can expect to see take place in 2025.

Tile-based Logs are the Future

After years of working with transparency logs at scale, the transparency.dev community has found there are better ways to make logs more scalable and cheaper to operate. A new approach ‘Tile-based Transparency Logs’, built on the concept of exposing storage as “tiles”, has already been tested and proven in several ecosystems since 2021. 2024 was a transformative year for tile-based logs, beginning with the release of the Sunlight log by Let’s Encrypt. This was followed by a community developed standardized API for CT tile-based logs known as the Static-CT API, which rapidly became the foundation for new and experimental log implementations (e.g. itko). The year concluded with the alpha release of Tessera, a general-purpose tile-based log developed by the creators of Trillian, further cementing the viability of this innovative approach.

The Chrome CT team have already shared plans to start accepting tile-based Static-CT API logs into the ecosystem, signalling a significant evolution in Certificate Transparency. Introducing major changes to a mature ecosystem is a huge undertaking and can often be highly disruptive. For context, RFC 9162 was developed as an improved version of RFC 6962 but was never widely adopted as the incremental benefits did not justify the cost and complexity of implementation. This highlights the importance of tile-based logs, as their adoption demonstrates the CT ecosystem’s willingness to evolve when the benefits are compelling and transformative.

Cloud Native CT Logs

While next-generation transparency logs emphasize their tile-based architecture and API, it is equally important to highlight their cloud native approach to storage management. By natively supporting cloud object storage, these logs unlock significant advantages such as operational simplicity, high throughput and the ability to dynamically scale based on usage. Sunlight, for instance, supports S3-compatible object stores, which Let’s Encrypt leverages to reduce costs and scale up easily. In the case of Trillian Tessera, each storage backend is independently implemented and takes the most native approach for the infrastructure it targets. The Tessera alpha includes support for object storage on both GCP and AWS, showcasing its commitment to leveraging cloud native principles. By embracing cloud object storage, next-generation transparency logs reduce the complexities associated with traditional storage solutions. This shift enables organizations to benefit from high availability, dynamic scaling, and simplified maintenance, paving the way for more efficient and resilient transparency logging systems.

The Growth of Transparency Ecosystems

The CT ecosystem has inspired several other transparency-based ecosystems such as Sigsum for transparent key-usage and Key Transparency in Proton Mail,

In software supply chain security, Sigstore uses transparency logs to make it easer for developers to sign software and for users to verify its authenticity. In 2024, Sigstore saw significant adoption into popular OSS packaging ecosystems such as Python and Homebrew, demonstrating its practical application and broad impact. Python is moving towards using Sigstore exclusively for signing release artifacts, while Homebrew leverages Sigstore to provide verifiable provenance for its binary packages. Sigstore has evolved into being critical internet infrastructure that is relied upon by GitHub, GitLab, and other major software organizations across the planet. Looking ahead to 2025, Sigstore plan to transition to tile-based logs to take advantage of their lower operational costs and improved scalability.

In general as logs become easier to set up, maintain and operate, expect more industries to take advantage of transparency technologies. This trend could extend beyond traditional applications to emerging areas such as AI transparency (transparency in machine level models) and enhancing trust in digital identity ecosystems.

The Rise of Standards and Interoperability

With the rise of transparency adoption, there has been a parallel evolution of protocols and formats aimed at standardizing transparency systems. This development has led to the creation of more general-purpose and reusable building blocks. These evolving specifications, shaped through implementation, experimentation, and collaboration, have been consolidated in the The Community Cryptography Specification Project (C2SP) repository. Key specifications include:

• Generic Checkpoint - a spec for a signed note where the body is precisely formatted for use in transparency log applications

• Generic Tiles - a spec for an efficient HTTP API to fetch the signed checkpoint, Merkle Tree hashes, and entries of a transparency log containing arbitrary entries.

• Generic Witnessing - a spec for a synchronous HTTP-based protocol to obtain cosignatures from transparency log witnesses.

These specifications significantly improve interoperability not only within Certificate Transparency (CT) but also across different transparency ecosystems. This growing standardization enables the creation of shared horizontal services that can be utilized across various domains. A prime example of such a service is witnessing. Witnessing plays a crucial role in maintaining the append-only, globally consistent, and verifiable properties of transparency logs. The ArmoredWitness network made a significant advancement in this area by launching the first sustainable, operational network capable of supporting multiple logs, regardless of their payloads and will continue to grow and evolve in 2025.

The Road Ahead

The future of Certificate Transparency and the Transparency.dev ecosystem is bright. As more organizations recognize the importance of transparency in securing digital communications, the demand for robust CT solutions will only grow. While there is much to look forward to in 2025, several challenges remain, not least the looming challenge of adoption post quantum cryptography.

The Transparency.dev community will continue to play a pivotal role in shaping this future. By fostering collaboration, sharing knowledge, and building innovative tools, the ecosystem will empower security professionals and developers to make the digital worlds a safer place for everyone. Join us as we build a more transparent, accountable, and secure digital world!

• Join the transparency-dev Slack

• Follow the Transparency-dev Community Blog

• Learn more about the Transparency.dev Roadmap

Why PostgreSQL?

PostgreSQL is a powerful, open source relational database known for its robustness, scalability, and extensibility. Sectigo originally used MariaDB as the transparency log storage backend. However, a CT log failure earlier this year due to MariaDB corruption after disk space exhaustion provided the motivation for a change. PostgreSQL, with its robust Write-ahead Logging (WAL) and strict adherence to ACID (Atomicity, Consistency, Isolation, Durability) principles, made it a better option for avoiding corruption and improving data integrity of the log.

While reliability was the catalyst for this transition, other factors also motivated the shift. The Sectigo team has deep expertise with PostgreSQL, including extensive experience operating crt.sh for years. Their experience, which included architecting several iterations of crt.sh’s CT log ingestion application to accommodate the rapid growth of WebPKI issuance rates, proved invaluable when optimising write performance in this new Trillian feature.

Trillian + PostgreSQL

Open source thrives on collaboration and contributions and the transparency.dev community embodies this. Trillian welcomes contributions and using this process Sectigo opened a PR and worked with Google’s TrustFabric team to incorporate the changes, including tests and documentation, into Trillian. The Feature Implementation Matrix highlights PostgreSQL as one of the supported options now, currently in Alpha stage. Sectigo is working on spinning up development logs and will spin up new production logs (code name Elephant) in due course and looks forward to keeping the community up to date.

Getting Started with Trillian and PostgreSQL for CT

To take advantage of using PostgreSQL for CT:

1. Install the latest versions which support PostgreSQL: users will require upgrading to

   • Trillian v1.7.0

   • certificate-transparency-go v1.3.0

2. Set up PostgreSQL.

3. Configure a PostgreSQL instance with the appropriate schemas and permissions as needed

4. See Trillian deployment examples which now include a PostgreSQL docker-compose example.

Whether you’re setting up your first CT log or scaling an existing one, you now have more choices than ever and can now leverage PostgreSQL’s reliability and performance.

Key Features

1. Tiles API - Trillian v1 uses tiles in its internal storage, but does not expose them via its API and instead serves pre-built proofs. Trillian Tessera fully embraces the tiles concept by introducing a Tiles API, allowing clients direct access to subtree tiles from the log. This API unlocks the full potential of tile-based architecture and has many benefits including improved cacheability, and making logs easier to spin up and maintain. Ultimately, the tiled log architecture and API significantly reduce the total cost of operation compared with Trillian v1.

2. Multi-environment Support: Trillian Tessera has native implementations for on-premises and cloud environments. Initial support is provided for running with AWS, Google Cloud Platform (GCP), POSIX-compliant file systems, or MySQL databases. Tessera takes the most native approach for infrastructure it targets, with each storage implementation implemented independently. This enables a new level of performance and flexibility that ensures that no matter your infrastructure, Tessera can be deployed and adapted to a wide range of use cases.

3. High Write Throughput: Tessera was built with speed in mind. The refined design separates sequencing and integration of entries to allow for higher write throughput and enables applications to be built with better practices, such as offering offline-proof bundles.

4. Bring-Your-Own Log Personality: Trillian Tessera enables building of customizable log personalities, including full support for the unique requirements of static CT API-compliant logs.

5. Resilience and Availability: Tessera’s design allows for multiple write frontends to run concurrently on the same storage, ensuring high availability and resilience, and reads to be served directly by storage infrastructure. This setup provides safety measures against accidental misconfigurations, such as launching multiple instances with the same storage settings, offering greater reliability compared to Trillian v1 logs on similar infrastructure.

Tessera Performance

Tessera is designed to scale to meet the needs of most currently envisioned workloads in a cost-effective manner. All storage backends have been tested to meet the write-throughput of CT-scale loads without issue. The read API of Tessera based logs scales extremely well due to the immutable resource based approach used, which allows for:

• Aggressive caching to be applied, e.g. via CDN

• Horizontal scaling of read infrastructure (e.g. object storage)

Exact performance numbers are highly dependent on the exact infrastructure being used (e.g. storage type). During our alpha testing phase, we ran extensive benchmarks using Hammer, our load testing tool for Tessera logs. This was done to ensure Tessera could handle the demands of high-throughput environments, while ensuring the logs were behaving correctly.

We have captured a snapshot of performance metrics for Trillian Tessera storage implementations under different conditions. Additionally, we documented the methodology for users to recreate these experiments independently. For further details, refer to Tessera Storage Performance.

Get Started

Trillian Tessera is a Go library designed to make it easy to build and deploy new transparency logs on supported infrastructure. To get started, check out the accompanying Conformance Codelab, which provides a step-by-step guide for setting up your first log, writing some entries to it using HTTP, and inspecting its contents with ease.

We’ve made it easy to start experimenting with Tessera by providing these example personalities:

1. POSIX: example of operating a log backed by a local filesystem. This example runs an HTTP web server that takes arbitrary data and adds it to a file-based log.

2. MySQL: example of operating a log that uses MySQL. This example is easiest deployed via docker compose, which allows for easy setup and teardown

3. AWS: example of operating a log running on AWS. This example can be deployed via terraform, see the deployment instructions.

4. GCP: example of operating a log running in GCP. This example can be deployed via terraform, see the deployment instructions.

For more information on getting started and to stay updated with future examples, visit the Getting Started section in the Tessera README

Get Involved in the Alpha

As we continue to refine and enhance Trillian Tessera, we invite developers and organizations to start using the alpha. Your feedback on Tessera’s performance and usability is crucial as we work towards a full release. By trying out the alpha, you’ll have the opportunity to influence the direction of development and ensure it meets the needs of the community. We also welcome contributors - check out our contributing guide for more details.

What’s Next?

Looking forward, we have ambitious plans for Trillian Tessera. Future plans will focus on adding features that enhance usability and integration with other tools, especially based on feedback we hear from our early adopters. We are developing a static CT API compatible personality built with Tessera, anticipated in early 2025. Additionally, we are committed to ensuring that existing Trillian users have a smooth migration path to Trillian Tessera logs.

Stay tuned for updates, and if you’re interested in getting involved check out the Tessera Github repository and sign up to the Transparency.dev Slack.

This release marks an important milestone for the Transparency.dev community as tiled-logs become available for any transparency ecosystem. Thank you for your support, and we can’t wait to see what you build with Trillian Tessera!

Breaking down these properties:

• Verifiable: you can cryptographically prove that the data hasn’t been unexpectedly changed

• Globally consistent: whoever comes to the log, wherever they are in the world, and whoever they happen to be, sees the same list of the data in the log

• Append-only: once data is added to the log it can neither be deleted nor modified; new data can only be added to the end of the log

So how do we ensure that these properties are being met in a transparency system? How do we make sure our verifiable systems are actually verified?

Why Witnessing Matters

This is the exact problem that witnesses were invented to solve. A witness is an entity that implements a gossip protocol to effectively verify that logs are append-only, ensuring their integrity and preventing unauthorized modifications. When operating in coordination with other such entities, witnesses help safeguard against split view attacks by maintaining global consistency for all logs. This collaborative approach ensures that all log users have a unified, tamper-resistant view of the log’s history.

How do Witnesses Work?

A witness cosigns checkpoints consistent with those it previously signed. A witness gets a checkpoint from a transparency log, and signs it only after verifying the consistency proof, ensuring the checkpoint is consistent with any previous checkpoints it has signed. By doing so it asserts the append-only security property of the log. Only a single witness is required to prove a log is append-only.

A set of witnesses (ideally an odd number) can be used to detect split view attacks and help reliably prove that a log is globally consistent. The more witnesses you can trust who have seen the same thing, the more you can trust that the view of the log. Strengthen that further with a finite pool of witnesses and then using a quorum technique N of M to guarantee that the log is globally consistent.

It is important to note that witnesses are not concerned with the contents of the log's leaves. For example for a firmware transparency log, a witness would not be able to validate legit firmware was logged. Verifiers such as auditors or monitors are used to ensure log contents are legitimate. Such verifiers are bespoke to a specific ecosystem, unlike witnesses which are designed to witness almost all logs. Witnesses underpin these verifiers to ensure that the same view of the contents is presented to every log user.

Evolution of the Witness Network

OmniWitness

The first significant implementation of a witness network was the OmniWitness network. Developed by the TrustFabric team, the OmniWitness network monitored several logs that use the generic checkpoint format. These logs include:

• Go SumDB

• Sigsum

• Sigstore

• Android Binary Transparency

• LVFS

• Armored Witness

A number of participants helped run the OmniWitness network and get it working. However one significant problem that arose was effectively handling policy updates (e.g. add a new log to the monitor list) and software updates, to ensure they were more deterministic. Nevertheless it was an invaluable way to kick start the concept.

ArmoredWitness

To create a more scalable and viable witness network, ArmoredWitness introduces a specialized, lightweight, plug and play solution that custodians around the world can easily install, verify and maintain. This iteration builds on the strengths of the OmniWitness software, integrating it with a custom, open source hardware chip and tailored packaging to streamline deployment.

The ArmoredWitness was engineered to be fully transparent, open to inspection and verification. Key features include:

• Open source hardware and software

• Reproducible builds with a firmware transparency log: updates are easily managed and traceable, establishing clear, auditable provenance.

• Full boot-chain verification from mask ROM to running software

Currently, 15 ArmoredWitness devices are deployed with trusted custodians, collaboratively bootstrapping a diverse, tamper-resistant witness network. This network not only offers greater resiliency and minimizes risk of team coercion, but also serves as a living example of fully-end-to-end firmware transparency in action.

Looking Forward: The Future of Witness Networks

As the adoption of transparency logs grows, implementing a comprehensive set of transparency roles to uphold core transparency principles is more important than ever. Witnessing plays a key role in this ecosystem, underpinning the append-only, globally consistent and verifiable properties. The ArmoredWitness network advances the field by offering the first sustainable, operational network capable of supporting multiple logs, regardless of payload.

Looking forward, recent community discussions focussed on a number of areas including:

• Driving adoption of the ArmoredWitness network

• Addressing scalability challenges such as configuration and format standardization

• Establishing robust policy frameworks

• Increasing number of witnesses as well as reputational units in the ecosystem. (TrustFabric, as the operators of the ArmoredWitness, are the reputational unit behind the current network.) The more independent reputational units running witness networks, the more the ecosystem is strengthened.

By tackling these challenges, we can build towards the goal of ensuring resilient, scalable witness networks that reinforce transparency and trust across log ecosystems.

This blog post is based on the talk ‘ArmoredWitness, putting verifiable transparency in your verifiable transparency’ by Martin Hutchinson.

Evolution of the Bar for Trust

The summit kicked off with an inspiring keynote from Al Cutter who took us through the fascinating evolution of transparency technologies. He shared how these technologies have redefined trust in the industry and underscored the growing importance of transparency as the gold standard. Cutter also highlighted C2SP specs which are driving interoperability across ecosystems. Cutter’s insights set a powerful tone for the rest of the summit, framing the discussions around how we can drive further innovation through collaboration in open source.

Tiled Logs are the Future

Matthew McPherrin shared the operator pain of logs in his talk about Let’s Encrypt’s experience and the evolution of the Sunlight log. Sunlight has led to the development of the standardized static-ct API. Al Cutter and Martin Hutchinson also introduced Trillian Tessera, a new tiled based log that supports the static-ct API but also works for any transparency log. Hayden Blauzvern gave an overview of Sigstore and highlighted their efforts to capitalize on tiled logs, as they experiment with Trillian Tessera with the goal of enhancing simplicity and reducing costs.

A Witness Network is Coming

Andrea Barisani, Filippo Valsorda, Martin Hutchinson and Al Cutter came together to explore the journey from the foundational concept of “witnessing” to the realization of fully transparent, end-to-end ecosystems. Their four talks interlocked seamlessly to provide a holistic view of witnessing and its broader impact.

Certificate Transparency Continues to Grow and Evolve

Joe DeBlasio and Philippe Boneff spoke about the challenges transparency ecosystems run into when they scale, and highlighted that CT is going strong, and is a great model for other ecosystems. CT continues to evolve and address challenges such as adopting post-quantum cryptography.

Transparency is Successful, the Community is Awesome!

Filippo Valsorda emphasized the community’s success, noting how it has broken out of its niche and reached a broader audience. This progress was further reflected in the diversity of the talks, such as Edona Fasllija's presentation on digital identity, Peter Well’s overview on using transparency in healthcare, and many others, showcasing the wide range of topics now engaging the community. Key transparency and binary transparency were also important topics for discussion.

The summit was a great way for the community to connect and have fun.

“This community is filled with people that want to do the right thing” - Joe DeBlasio, Chrome

“Huge thanks to the #trustfabric and all the org team members for the hard and wonderful work. And to the warm Transparency community for its capacity to welcome new members even if they are not core contributors (by far!)” - Christophe Brocas, Assurance Maladie security engineer

“My favorite workshop of the year.” - Bas Westerbaan, Cloudflare

If you missed the event or want to stay in touch with the community for future events, check out these links:

Also stay in touch with the community through these channels:

• Join the transparency-dev Slack

• Watch the talk recordings on the transparency-dev YouTube channel

• Follow the Transparency-dev Community Blog

Why Tiles?

In this article titled Tile-Based Transparency Logs, Jay Hou outlines how the tiles approach works and has been tested and proven in ecosystems such as the Go module checksum database and Pixel binary transparency. The article also outlines the many benefits of a tiles-based architecture:

• Improved cacheability

• Logs are easier to spin up

• Logs are easier to maintain

• Static responses

• Uniform interface across logs and ecosystems e.g. witness network can be shared across all logs

Why Tessera?

The name “Tessera” was chosen to reflect the core concept of the new design. A tessera is a small square tile of stone, glass, marble etc. that was used in mosaics. This is an apt metaphor for our project, as it emphasizes the tile-based structure of transparency logs.

Trillian Tessera Design Philosophy

While a tiles-native API is central to the design, we also incorporated several insights gained over the years. These insights have informed our key design philosophy.

• Simplicity - Tessera is designed to be user-friendly, easy to adopt, and straightforward to maintain. It offers a more cost-effective and simpler operation compared to Trillian v1. As a library rather than a microservice architecture, Tessera aims to facilitate the easy creation and deployment of new transparency logs on supported infrastructure.

• Multi-implementation Storage - Each supported storage infrastructure for Trillian Tessera is independently implemented, and takes the most "native" approach for the targeted infrastructure. This includes support for both cloud and on-premises environments. Initially we will provide support for GCP and AWS. Additionally, Tessera will offer cloud-agnostic support for MySQL and POSIX file systems, making it cloud-agnostic and scalable. This allows Tessera to accommodate a range of transparency log use-cases, from high-performance, high-uptime logs like Certificate Transparency (CT) and Sigstore, to those with more modest requirements, such as Firmware Transparency.

• Fast Sequencing and integration of Entries - In Trillian, entries were added to a log via a fully decoupled queue, providing a timestamp and a future promise of integration. With Trillian Tessera, we have refined the storage contract so that calls to add entries to the log will return with a durably assigned sequence number or an error, with integration occurring within seconds. This separation of sequencing and integration allows for higher write throughput and enables applications to be built with better practices, such as offering offline-proof bundles.

Trillian Tessera Status

Trillian Tessera is currently under active development, and is not yet ready for general use. However, early feedback is very welcome. We expect an alpha release by Q4 2024, and production ready anticipated in the first half of 2025. We are also developing a CT static API compatible server built with Tessera, which is expected to be released on a similar timeline. Trillian v1 is still in use in production environments by multiple organisations in multiple ecosystems, and is likely to remain so for the mid-term. New ecosystems, or existing ecosystems looking to evolve, should strongly consider planning a migration to Tessera and adopting the patterns it encourages.

Get Started

You can learn more about Trillian Tessera and try it out here: https://github.com/transparency-dev/trillian-tessera

Certificate Transparency, now in its 11th year, has led the way making the issuance of website certificates transparent and verifiable. Transparency technology has been adopted in various other domains including software supply chains, firmware transparency and binary transparency. Transparency technology continues to evolve to be easier to use and maintain as well as work in multiple use-cases. The evolution and adoption of transparency technologies reflects a growing emphasis on accountability and trust in our digital ecosystems.

The Transparency.dev Summit aims to bring together implementers, operators, and clients of real world transparency systems in order to meet peers, share best practices, and learn about the latest developments in the community. In addition to general transparency topics, the event will also feature dedicated sessions for the Certificate Transparency community.

Attendees will have the opportunity to learn more about transparency systems at scale and the latest developments in the transparency ecosystem. Some central themes of the conference will be:

• Witnessing: A key goal for transparency is to have a cross-ecosystem witness network that can provide protection to prevent split-view attacks. Talks here will examine the latest developments in the witness network and related technologies.

• Certificate Transparency: This will highlight lessons learned from log operators, especially those that can be applied to newer logs and ecosystems looking to emulate the success CT has achieved for over 10 years.

• Key Transparency: This emerging space aims to solve the problem of key discovery for providers of end-to-end encryption services.

• Binary Transparency: This is a wide umbrella term for bringing enhanced security to language ecosystems e.g. with Sigstore to firmware transparency and more

• Transparency futures: learn about the new and recent evolutions in transparency technologies that we can look forward to that make logs easier to run and maintain.

Some of the Trust Fabric team who initiated the conference share their thoughts about the event:

“Transparency.dev Summit, dedicated to real-world transparency systems, is the first event of its kind. After much anticipation, the perfect conditions and growing interest in transparency technologies have come together to make this event a reality. We’re eager to gather the entire community, to connect, share, and see what incredible things we can achieve together.” - Al Cutter, recent Levchin Prize Winner.

“I’m looking forward to seeing everyone and having meaningful conversations about transparency technology, continuing where we left off at CATS. I’m also excited to meet new faces and explore fresh ideas together!” - Martin Hutchinson

“I hope that I can go!” - Jay Hou

If you are interested in submitting a proposal or would like to learn more about the event, please visit transparency.dev/summit for the latest information. Registration is not yet open though you can register your interest here: Interest Survey.

We hope you can join us and be part of the conversation driving the future of security and trust in our digital worlds through transparency technologies.

The Significance of the Levchin Prize

The Levchin Prize for Real-World Cryptography, named after Max Levchin, is awarded annually to individuals and projects that have made substantial contributions to practical cryptography. The prize recognizes achievements that have had a profound impact on the security of the digital world. Past winners include the OpenSSL Team, Ralph Merkle and Let's Encrypt.

Awarding the Levchin Prize to Certificate Transparency acknowledges its transformative effect on internet security. By ensuring that all certificates are publicly visible and verifiable, CT has drastically reduced the risk of certificate-related attacks, making the web a safer place for users and businesses alike.

“We Wanted To Give Up”

The award was announced at the Real World Crypto 2024, chaired by Dan Boneh. Emilia Käsper, Adam Langley, and Ben Laurie were present at the ceremony to receive the award.

In her acceptance speech, Emilia Käsper looked back at the CT journey starting with the launch of the first CT log, Pilot, in 2013. Emilia revealed that less than a year in, the team could not get the design to work with the real life constraints and were on the verge of giving up. Despite the challenges the team did not give up, so how did they push through? By believing there had to be a way.

The Next Challenges

Ben Laurie followed up by sharing the next things we should all be believing in and focussing on. He says “The bad news is that the real work of real-world crypto is not crypto, it’s actually user-experience, ecosystem change and systems engineering.”

Ben also highlighted two really huge problems to address in Public Key Infrastructure: Time and Identity. You can watch the whole award ceremony here:

The Power of Community

Ben also acknowledged in his speech that CT took dozens, if not hundreds, maybe even thousands of people to get it deployed and thanked all of them too. Likewise, Al Cutter who was the first engineer on CT and still head of the engineering team that supports CT at Google said: “This recognition also reflects onto everyone who played a part in making CT a reality.”

Congratulations to the entire community on this well-deserved achievement! 👏👏👏