
Indie Hacker | ⭐️ hacking TRACARDI - Open Source Customer Data Platform. #nocode #ecommerce #opensource #customerexperience #CX #headless #cdp
Poland
Joined April 2021
- Tweets985
- Following562
- Followers331
- Likes997
Missing Link between AI and Knowledge linkedin.com/pulse/missing-l… via @LinkedIn
13
Building LLMs that remember?
I’ve launched a subreddit for developers and researchers working on long-term memory architectures for LLMs — retrieval, embeddings, and beyond.
Join here → reddit.com/r/llm_memory/
31
Welcome to the wilds of 2025, where the landscape is constantly shifting, AI is running the show… link.medium.com/pRWXJroZVPb
71
The end of the year is approaching, and it’s time for reflections and summaries. Tracardi has achieved another successful year with an impressive organic growth of over 40%. This growth was accomplished without the involvement of investors or marketing efforts.
1
61
I like it. Show me only open-source Customer Data Platforms. Only #tracardi on the list.
43
Risto | Tracardi retweeted
Most people think the cloud saves them money.
Not with Kafka. ❌
Storage costs alone are 32 times more expensive than what they should be.
Even a miniscule cluster costs hundreds of thousands of dollars!
Let’s run the numbers. 🤓
Assume a small Kafka cluster consisting of:
• 6 brokers
• 35 MB/s of produce traffic
• a basic 7-day retention on the data (the default setting)
With this setup:
1. 35MB/s of produce traffic will result in 35MB of fresh data produced. 👌
2. Kafka then replicates this to two other brokers, so a total of 105MB of data is stored each second - 35MB of fresh data and 70MB of copies 🤔
3. a day’s worth of data is therefore 9.07TB (there are 86400 seconds in a day, times 105MB) ✅
4. we then accumulate 7 days worth of this data, which is 63.5TB of cluster-wide storage that's needed 😳
Now, it’s prudent to keep extra free space on the disks to give humans time to react during incident scenarios, so we will keep 50% of the disks free.
☝️Trust me, you don't want to run out of disk space over a long weekend.
63.5TB times two is 127TB - let’s just round it to 130TB for simplicity.
That would have each broker have 21.6TB of disk. 💡
On to Pricing 🤑
We will use AWS’s EBS HDDs - the throughput-optimized st1s.
Note st1s are 3x more expensive than sc1s, but speaking from experience... we need the extra IO throughput.
Keep in mind this is the cloud where hardware is shared, so despite a drive allowing you to do up to 500 IOPS, it's very uncertain how much you will actually get. 💡
Further, the other cloud providers offer just one tier of HDDs with comparable (even better) performance - so it keeps the comparison consistent even if you may in theory get away with lower costs in AWS. 👌
st1s cost 0.045$ per GB of provisioned (not used) storage each month. That’s $45 per TB per month. 💡
We will need to provision 130TB.
That’s:
🛑 $188 a day 😕
🛑 $5850 a month 😳
🛑 $70,200 a year 🤯
btw, this is the cheapest AWS region - us-east.
Europe Frankfurt is $54 per month which is $84,240 a year. 😭
But is storage that expensive?
Hetzner will rent out a 22TB drive to you for… $30 a month.
6 of those give us 132TB, so our total cost is:
🟩 $5.8 a day
🟩 $180 a month
🟩 $2160 a year
Hosted in Germany too.
AWS is 32.5x more expensive! ❌
39x times more expensive for the Germans who want to store locally. 🇩🇪😥
Let me go through some potential rebuttals now. 🙅♂️
🙅♂️ What about Tiered Storage?
It’s much, much better with tiered storage. You have to use it.
It'd cost you around $21,660 a year in AWS, which is "just" 10x more expensive. But it comes with a lot of other benefits, so it's a trade-off worth considering.
I won't go into detail how I arrived at $21,660 since it's a unnecessary.
Regardless of how you play around with the assumptions, the majority of the cost comes from the very predictable S3 storage pricing. The cost is bound between around $19,344 as a hard minimum and $25,500 as an unlikely cap. 👌
That being said, the Tiered Storage feature is not yet GA after 6 years... most Apache Kafka users do not have it.
🙅♂️ What about other clouds?
In GCP, we'd use pd-standard. It is the cheapest and can sustain the IOs necessary as its performance scales with the size of the disk.
It’s priced at 0.048 per GiB (gibibytes), which is 1.07GB.
That’s 934 GiB for a TB, or $44.8 a month.
AWS st1s were $45 per TB a month, so we can say these are basically identical. 👍
In Azure, disks are charged per “tier” and have worse performance - Azure themselves recommend these for development/testing and workloads that are less sensitive to perf variability.
We need 21.6TB disks which are just in the middle between the 16TB and 32TB tier, so we are sort of non-optimal here for our choice.
A cheaper option may be to run 9 brokers with 16TB disks so we get smaller disks per broker. 💡
With 6 brokers though, it would cost us $953 a month per drive just for the storage alone - $68,616 a year for the cluster. (AWS was $70k)
Note that Azure also charges you $0.0005 per 10k operations on a disk.
If we assume an operation a second for each partition (1000), that’s 60k operations a minute, or $0.003 a minute.
An extra $133.92 a month or $1,596 a year. Not that much in the grand scheme of things.
If we try to be more optimal, we could go with 9 brokers and get away with just $4,419 a month.
That’s $54,624 a year - significantly cheaper than AWS and GCP's ~$70K options.
But still more expensive than AWS's sc1 HDD option - $23,400 a year.
All in all, we can see that the cloud prices can vary a lot - with the cheapest possible costs being:
• $23,400 in AWS
• $54,624 in Azure
• $69,888 in GCP
Averaging around $49,304 in the cloud.
Compared to Hetzner's $2,160... 💀
🙅♂️ Can Hetzner’s HDD give you the same IOPS?
This is a very good question.
The truth is - I don’t know. 🤷♂️
They don't mention what the HDD specs are.
And it is with this argument where we could really get lost arguing in the weeds. There's a ton of variables:
• IO block size
• sequential vs. random
• Hetzner's HDD specs
• Each cloud provider's average IOPS, and worst case scenario.
Without any clear performance test, most theories (including this one) are false anyway. 🤷♂️
But I think there's a good argument to be made for Hetzner here.
A regular drive can sustain the amount of IOs in this very simple example. Keep in mind Kafka was made for pushing many gigabytes per second... not some measly 35MB/s.
And even then, the price difference is so egregious that you could afford to rent 5x the amount of HDDs from Hetzner (for a total of 650GB of storage) and still be cheaper.
Worse off - you can just rent SSDs from Hetzner! They offer 7.68TB NVMe SSDs for $71.5 a month!
17 drives would do it, so for $14,586 a year you’d be able to run this Kafka cluster with full on SSDs!!!
That'd be $14,586 of Hetzner SSD vs $70,200 of AWS HDD st1, but the performance difference would be staggering for the SSDs. While still 5x cheaper.
🙌 Pro-buttal: Increase the Scale!
Kafka was meant for gigabytes of workloads... not some measly 35MB/s that my laptop can do.
What if we 10x this small example? 60 brokers, 350MB/s of writes, still a 7 day retention window?
You suddenly balloon up to:
• $21,600 a year in Hetzner
• $546,240 in Azure (cheap)
• $698,880 in GCP
• $702,120 in Azure (non-optimal)
• $700,200 a year in AWS st1 us-east
• $842,400 a year in AWS st1 Frankfurt
At this size, the absolute costs begin to mean a lot.
Now 10x this to a 3.5GB/s workload - what would be recommended for a system like Kafka... and you see the millions wasted.
And I haven't even begun to mention the network costs, which can cost an extra $103,000 a year just in this miniscule 35MB/s example.
(or an extra $1,030,000 a year in the 10x example)
More on that in a follow-up.
In the end?
It's still at least 39x more expensive.
If you think this problem deserves more attention - consider pressing the heart button and sharing to your network. 🙏
How to Contribute to Tracardi Q&A
blog.tracardi.com/how-to-con…
1
27
Risto | Tracardi retweeted
ChatGPT Prompt Framework:
Follow @heyDhavall for more!
🌟 Exciting news! We're thrilled to introduce the Tracardi blog! 🎉 Dive into insights on customer data, personalization, and automation strategies that can transform your marketing game. 🔑💡 #Tracardi #CustomerExperience #DataDrivenMarketing
blog.tracardi.com/
Risto | Tracardi retweeted
A Kafka in the cloud doing 30MB/s costs more than $110,000 a year.
A $1,000 laptop can do 10x that.
Where did we go wrong? 👇
The Cloud. Namely - its absurd networking charges 👎
Let’s break it down simply:
• AWS charges you $0.01/GB for data crossing AZs (but in the same region).
• They charge you on each GB in and out. Meaning each time a GB passes, you pay twice - for the one who sends it (outgoing) and the one who receives it (incoming)
• For a normal Kafka cluster with replication factor of 3 and a read fanout of 3x, you are going to be charged:
• 2x for 2/3rd of the produce throughput
• 4x for 100% of the produce from replicating it
• 6x of 2/3rd of the produce throughput for consumption.
(but it can get a lot worse - read until the end to see)
Simple example:
• 3-broker cluster, each in a separate AZ
• 3 producers, each in a separate AZ
• 3 consumer groups with 3 consumers each, each group with consumers in a separate AZ
The producers are producing 30MB/s in total to the same leader.
2/3 producers are in a different AZ, so 20MB/s of produce traffic is being charged at cross-zone rates. 👌
It’s charged both on the OUT (producer’s side) and IN (broker’s side).
The leader is replicating the full 30MB/s to both of its replicas.
This is again being charged both on the OUT (leader’s side) and IN (follower’s side), for both replication links. (60MB/s)
Then, each of the 3 consumer groups has 3 consumers.
All consumers read from the leader, with 2/3 in a different zone.
This results in 20MB/s of consume traffic charged at cross-zone rates PER GROUP. (60MB/s total)
Again charged both on the OUT (broker’s side) and IN (consumer’s side).
The total amounts to 140MB/s worth of cross-AZ traffic. Charged both ways.
When one MB is $0.00001/s, this means we’re paying $0.0028/s. 🤔
That’s:
• $241 a day 😕
• $7500 a month 😥
• $88,300 a year 🤯
It all goes down the drain on network traffic ALONE. 🔥
What about the hardware?
Quick napkin math assuming:
• 7 day retention
• all of the data is on EBS (not using tiered storage since it's not GA yet)
• keeping 50% of the disk free for operational purpose (don't ask me what happens if we run out of disk)
• the 3 brokers are running modest r4.xlarge instances (kinda overkill but hey, why not)
We'd pay:
• $19,440/yr for the EBS storage
• $6,990/yr for the EC2 instances
That’s right - you’re paying just $26.4k/yr for the hardware and 88.3k for the network (3.3x the hardware)
For a total of $115k/yr. 💸
I’m not even counting load balancer costs, which could be $12k by some quick napkin math too.
How ridiculous is that? 😂
Want it to get more ridiculous?
This calculation assumes you’re hosting your own Kafka cluster in the same AWS account.
💡If you use a managed Kafka provider that’s not AWS, or otherwise just another AWS account, you’re typically connecting to them through a public endpoint.
AWS then charges all traffic at the cross-AZ $0.01/GB rate internet traffic rate, even if it's in the same AZ.
The end result?
$113,000 a year for network costs. 💀
For 30MB/s. (!!!)
btw - 30 MB/s is absolutely nothing for Kafka... 🤡
It is most often network/disk bounded.
Doing 3GB/s is not hard. 👌
The higher throughput you go, the more absurdly large this discrepancy between network and hardware cost becomes.
For example - this exact setup could probably do 3x the traffic (90MB/s), assuming storage space isn't a concern.
Then you'd have:
• $264,000 a year for the cross-AZ rate. 🥲
• $339,000 a year for the internet rate. 💀
Why is this cost (more than 300k a year) and complexity (this calculation) the case when three laptops can run this practically for free?
Where did we go wrong?
Worth Noting:
There are a few optimizations that can be done here:
• consumers can use fetch from follower, which results in free read traffic (no cross-AZ charges) in the first example. But the second example would still be charged internet costs. 🤝
• you can avoid internet costs by VPC-peering or Private Link-ing the two AWS accounts. This is largely what most cloud providers do, otherwise it becomes prohibitively expensive. It can be super complex to do. 🔧
• AWS can give you large discounts (up to 90%+ afaict) on the quoted prices, depending on your usage. It’s unclear what customer gets what discount. 💰
And perhaps the best example - you can use an ingeniously-designed product like WarpStream that eliminates all of this complexity and cost. ⭐️
It's no wonder they got acquired after just 13 months of operation.
Modern Marketing Pipelines and CDP
blog.tracardi.com/supercharg…
1
27








