Ergodic's Causal Model, W, vs. CDFM
How W, our model for finding cause and effect in a table, compares with CDFM and the methods the field has relied on for years.
W is Ergodic's model for working out cause and effect from a table of data. On synthetic benchmarks, it's the strongest causal-discovery model we've tested.
With its release, we want to show how it compares with CDFM, the strongest public model of its kind, and with the methods the field has relied on for years. On test problems built so that the right answer is known in advance, W was more accurate overall than every other method we tried. On real-world data, W matches CDFM at finding which things are connected.
Why causal discovery is hard
Most data tools find correlation: two things that tend to move together. Causal discovery asks a harder question. Looking only at a table of observations, can you tell which things drive which? Rain and umbrellas show up together, but the numbers alone don't say that rain brings out the umbrellas and not the other way round.
The answer comes in two parts. First, which variables are connected at all? Second, which way does each connection run: does A drive B, or does B drive A? Part two tends to be the harder one, because a correlation looks the same from both ends.
The two parts of a causal answer. A method only gets credit for a link when it finds the connection and points it the right way.
Older methods, such as LiNGAM, PC and GES, each make one assumption about how the data behaves, and they work well only when that assumption holds. Newer models like CDFM and W take a different route. They're trained in advance on a large number of example tables where the map of what causes what is already known (this map is called a graph). Shown a new table, they give an answer in one go.
How we tested
We gave W, CDFM and five other methods (AVICI, LiNGAM, NOTEARS, PC and GES) the same fixed set of test problems, drawn from three places. The first was the kinds of test used in CDFM's own paper: computer-generated problems, a physical light-tunnel experiment called Causal Chambers, and the Tübingen cause-effect pairs. The second was CausalArena, an independent collection of tests. The third was six public datasets from the real world.
The results
Each method gets a score between 0 and 1, and higher is better. To earn credit for a link, a method has to get the direction right as well as spot that two things are connected. This score is called directed F1.
W against CDFM on each benchmark. The column on the right is W's score minus CDFM's, so a positive number means W is ahead. The real-data row shows CDFM's score on seven datasets. W scored six of them, and on those six CDFM is ahead by 0.09.
Where W leads
W's clearest lead is on computer-generated problems, where the true answer is known and every method can be marked exactly. CausalArena has three sets of these, and W scores highest of every method on all three. On the 2,500 general problems it beats CDFM by 0.09. When we compared the methods problem by problem on identical tables, W came out ahead of every other method on that set, both at finding links and at getting their direction right. On the other two sets, GES was slightly ahead at finding links alone, by about 0.05, but W led once direction was counted.
The lead is not limited to one test. Across all 5,893 computer-generated problems we tried, W is ahead of CDFM by 0.07 and ahead of AVICI by 0.18. On ADIA, a set of 1,880 small tables we have also used during development, the lead is wider still: 0.73 against 0.57.
The lead also holds as the problems get bigger. It's there at every size, including tables with more than 50 variables, and it's widest where the problem is hardest. On tables with more than 20 variables, and on dense ones where each variable is linked to many others, W leads CDFM by about 0.08.
W against CDFM by number of variables and by how dense the graph is. n is the number of test cases.
W is also dependable. It returned an answer for all but 16 of the 5,920 computer-generated problems, and those 16 were tables larger than it's built to handle. The older methods managed far fewer: DirectLiNGAM returned results on 3,223 and GES on 2,347.
Problems each method returned an answer for, out of 5,920 computer-generated tables.
Where it's closer
On real data the contest is closer. Across the six real datasets, W is level with CDFM at spotting which variables are linked, before direction comes into it (0.58 each). On Sachs, a dataset on how proteins signal inside cells, W reported nine links and all nine match the reference answer, with no false ones. No other method did better at finding links on that table. On CSuite, a set of small diagnostic problems, W scored 0.93 for finding links, the second highest of any method we ran, behind only PC-KCI (0.99), a version of the older PC method. PC-KCI does well here because CSuite's tables are small, and because W is cautious and misses a few real links. PC-KCI doesn't hold up elsewhere: it returned a result for only 10 of CausalArena's 2,500 general problems.
The two are also level on CausalAssembly, which simulates a manufacturing line using real production data: 0.37 for W against 0.39 for CDFM on direction, and 0.63 against 0.59 on links.
W is also cheap to run. Scoring 16 real-data tests took one graphics processor (an L4 GPU) and about 20 US cents.
Why this all matters
For years, causal discovery has meant choosing a method whose assumptions fit your data, then waiting for it to run. W shows that a single model, trained once in advance, can beat those methods across thousands of varied problems and return an answer on almost all of them.
It also sets a new best result for models of this kind on CausalArena, an independent benchmark. W leads CDFM on all three of its test sets, 3,500 problems in total.
What comes next
Getting direction right on real data is the priority for the next version of W. After that come larger tables, and data that falls into categories instead of a continuous range of numbers. We also plan to keep testing on a wider set of real datasets.
We'd love for you to be part of our next steps. Try it on your own data at ergodic.dev.
If you're curious why we're building W and Welt in the first place, read the manifesto.
Run the decision before you commit to it.
Bring a decision. We'll discuss how to model it and what data a simulation would need.
Talk To An Expert