From CERN to Cledar
How we learned about the world and who we are within it
CERN is home to the world’s largest machine – the Large Hadron Collider (LHC). Physicists at CERN generate many petabytes of data smashing particles to reveal the origins of the universe. CERN is also the birthplace of something called the Internet.
Cledar’s founders – Hubert and Piotr – contributed to some of the greatest scientific discoveries of the 21st century. Read on to learn how Hubert and Piotr’s experiences at CERN shaped the vision for the Cledar of today.
The Globe of Science and Innovation at CERN, the European Center for Nuclear Research.
Chief Technology Officer
Chief Executive Officer
In the lead up to the LHC’s launch in 2008, much planning was needed to handle the resulting petabytes of data. Due to the nature of scientific projects, most teams operated in silos, focusing on their own research needs. There was a need for “big picture” thinking.
In 2002, Piotr joined the Academic Computing Center Cyfronet (Part of the AGH University of Science and Technology) to work on the newly started Computing Grid “CrossGrid”, which was part of a larger collaboration linked to the Worldwide LHC Computing Grid (WLCG) and CERN. Piotr set up the very first computing cluster in Central and Eastern Europe connected to the worldwide computing grid.
Piotr in front of the ATLAS LHC detector in Meyrin, Switzerland.
Hubert in front of the data room in building 190 in Meyrin. The first data from beam tests of the prototype TOTEM LHC detectors was consolidated and analysed there.
At the request of a family friend and professor at Warsaw University of Technology, Hubert traveled to CERN for the summer to help with data analysis on the TOTEM experiment at the Large Hadron Collider. Hubert prototyped detectors to handle the data generated by 40 million collisions per second. To ensure approval of the TOTEM project, Hubert had to make sense of the data.
While many teams had their own grid monitoring tools reporting healthy statuses, complaints were mounting about jobs not running or completing properly. Piotr was in charge of grid monitoring and operations, and developed an agent system to probe the grid and report back. This led to the disturbing revelation that only about 20% of the grid was, in fact, healthy (green), while the rest was coming back with various errors. While officially known as “Service Availability Monitoring”, CERN personnel colloquially referred to the SAM’s as “Piotr’s tests.” This achievement was like a badge of honor for a while, but it helped to foster the Cledar culture of questioning the fundamentals at every opportunity.
Nodes of the very first computing grid cluster at CERN made of desktop computers. Meyrin data center.
Totem Roman Pot detectors in the LHC. Hubert simulated the LHC running conditions to optimally use the hardware infrastructure and carried out the data analytics of the recorded data.
After the successful TOTEM project, Hubert was again recruited by CERN to “simulate the collider”. The physics team needed to be sure that the data from within the accelerator would be captured accurately, but without quite knowing what would happen. Hubert was essentially tasked with anticipating physics that had never been measured before.
Hubert and Piotr met in 2004, over a meal with mutual friends, where Hubert learned that Piotr was to blame for his computer processing jobs not coming back complete. They immediately hit it off, and it was not long before ideas for future collaboration began to emerge.
Restaurant R1 in Meyrin, Switzerland, where Piotr and Hubert used to have hot discussions on technological topics.
Hubert and Piotr in France, Sergy, Rue du Crêt de la Neige, close to Hubert's house.
The work and aggressive deadlines were occasionally interrupted by informal gatherings at cafés, late-night parties in French attics, and mountain hikes in the gorgeous countryside on either side of the France-Switzerland border. The surroundings and the culture created by the coming together of some of the world’s greatest minds served as the perfect breeding ground for the ideas and inspiration that would shape future events.
One of the key sources of inspiration were the regular adventures into the Swiss mountains. There, in the municipality of Meyrin, Piotr and Hubert learned the local word for a special type of gate that leads into the mountains. Cledar.
For 4 years, Hubert’s team used every data algorithm, optimization technique, and ML / neural network at their disposal to simulate scenarios (e.g., different beam conditions, various detector setups) and arrive at full-specification detector setup (detector distances to the beam, performance of silicon wafers, shapes of magnetic fields, etc.). The team also contributed significantly to designing LHC running conditions. . In fact, most of the data-related pitfalls and challenges were addressed in this way ahead of the actual data taking. This hard preparatory work allowed to get highly influential physics results literally within a week once the LHC started operation.
The professional preparatory work led by Hubert meant that the team of scientists avoided most of the data-related pitfalls and challenges that typically precede the capture of quality data. Instead, they benefited from high-quality results within a week of operations starting, thereby accelerating their journey to achieving their goals.
Prototype of the Roman Pot detector before installation in the LHC. The detector was designed to withstand the energetic LHC beams at a distance of a fraction of a millimeter.
Computing nodes in CERN Data Center in Meyrin. Data originating from all LHC collision points is consolidated here and made available for further analytics in Worldwide LHC Computing Grid.
Piotr’s team was responsible for ensuring that every different team at CERN was able to access the tools it needed to and that these tools and teams were connected to the vast technology infrastructure at CERN and that data was captured and processed in the right way. This enabling role was played in a time before virtualization and clustering were mainstream solutions, and before cloud software deployment techniques like containers. Piotr’s team thrived in developing technology solutions that enabled scientists to focus on achieving their goals, without being limited by technology.
There were two interlinked projects. The first, the Worldwide LHC Computing Grid (WLCG), focused on providing computational and data storage infrastructure for LHC experiments (Atlas, Alice, CMS, LHCb), spanning world-wide data centers. The other project was Enabling Grids for E-science in Europe (EGEE), which provided multidisciplinary scientific infrastructure (not only LHC but also in medicine, climate sciences, biochemistry, etc.).
In 2004 there were 30 data centers with a couple of thousand CPU cores. In 2008, there were over 300 data centers with over 50k CPU cores. Today it is one of the largest hybrid infrastructures with over 170 data centers and more than a million CPU cores. Piotr had to navigate the challenges associated with such growth.
By the time the LHC came back online properly in 2008, much of Piotr’s infrastructure efforts had been implemented and the grid was scalable to support the growth of CERN-based research for years to come. What had been known as “Piotr’s tests” officially became Service Availability Monitoring or SAM, and was integrated into a resource-brokering system that improved capabilities. Individual experiments could now exclude non-compliant nodes or sites, and the system was further integrated with automated alarms, notifications, problem-resolution tracking systems, and more. All in all, some estimate this solution to have saved up to 60m CHF yearly in budget.
With a sense of accomplishment for his years at CERN, Piotr decides to take an opportunity that would lead to the informal beginnings of Cledar as a company.
Computing nodes of the Worldwide LHC Computing Grid installed in Meyrin. Close up view.
In Autumn 2008, right after Hubert defended his PhD thesis, the LHC incurred serious damage as a result of unreliable electrical connections. The energy stored in the accelerator was comparable to that of a high-speed train and, unfortunately, this was accidentally released, causing devastating damage to a large section of the machine.
Just 10 days after being turned on, an explosion puts the LHC offline for many months.
Quote from Hubert: “From our perspective, we gained more time to get ready with our physics program.”
Hubert became Head of Analytics for the TOTEM physics team and coordinated analytics of physics related to proton scattering, which revealed three main data challenges: 1) magnetic field uncertainty in the particle collision area, 2) erratic beam positioning, and 3) excess background noise.
Solving these challenges enabled the most precise measurement of the proton structure. Hubert’s data set also enabled the discovery of a new particle (hadron), called an Odderon.
Large Hadron Collider installed in a tunnel spanning over the circumference of 26.7 kilometres under Switzerland and France.
After many successful projects and collaborations, Hubert felt it was time to apply his now-extensive expertise to the private sector and return to his native Poland. Under the leadership of Piotr and other CERN scientists, Cledar was ready for a full-time CEO who could lead new ML projects, build teams and create a healthy structure for the growth of the organization.
Hubert and Piotr’s experiences at CERN helped shape Cledar as a company. Today, Piotr continues to work with challenging data infrastructures and Hubert continues to conceptualize solutions in data science. With this core expertise and the addition of other experts, Cledar continues to grow, although Piotr and Hubert look back at the years at CERN with fondness and are grateful for the opportunities it created.
parties in French attics
late-night coffees in R1/R2/R3
challenges to overcome
discoveries left to find
Hubert and Piotr discussing the challenges and ideas for the future.
At Cledar, we practice what we preach. As an organization that builds data-driven solutions, we are committed to gathering data and feedback from our team to improve the way we work and communicate, and to understand what Cledar people want work on next. We listen, we analyze, we generate insight, and we act.