Above the patient’s body, suspended in the air, levitates his magnified, semi-transparent heart. Ideas straight out of science-fiction movies are becoming the reality of 21st century medicine before our very eyes
Embraced by the comfort of the chairs that fill the hall of the ICE Kraków Congress Center we are waiting for a showing to start. We are attending the New Frontiers in Interventional Cardiology (NFIC) workshops. In a moment, we are about to see a live broadcast of a heart surgery carried out by the experts from a cardiology clinic in Kraków.
“The patient has an atrial septal defect,” professor Dariusz Dudek, chairman of the Council of the Institute of Cardiology and head of the 2nd Clinic of Cardiology and Cardio-Vascular Interventions in Kraków, explains to the gathered cardiac surgeons. “The procedure will be carried out without opening the patient’s chest. The surgeon will introduce a catheter into the left atrium, then to the right one, after which he will install the closure device on the defective septum. We will show both how the procedure was planned thanks to the previously-obtained computer tomography, magnetic resonance, and ultrasound imaging, and what technology is used for monitoring the ongoing procedure. We will see the catheter move, close the heart’s structures, and effect the repair – and all of it in real time.”
The operating room appears on the screen. A man is lying on the table with green-clad physicians leaning over him. Around them, a few more members of the medical personnel are busying themselves with preparations for the procedure. It looks just like a film, except this isn’t a film at all. At this very moment, one kilometer away from where we are, the man and his heart are really waiting for help.
Heart in hand
The procedure begins. The lead surgeon puts a pair of special goggles on his head. They are the HoloLens with software adapted for use in cardiac surgery. Then the view changes and the screen shows us what the physician is seeing right now.
The sight is really astonishing. Above the patient’s body, suspended in the air, levitates his magnified, semi-transparent heart. It looks like a ghost. This image has been converted from echocardiography. It does not obscure reality, but rather it seems to have been inserted into reality. The surgeon extends his hand; for a moment, his fingers have difficulty to grab this strange holographic object, but after a moment he manages to grip it and turn it in the air to see better the defect in the heart’s septum.
I am having flashbacks to scenes from the “Star Wars” movies. It seems that ideas straight out of science-fiction movies are literally becoming the reality of 21st century medicine before our very eyes.
“The HoloLens are Microsoft’s wireless mixed reality smartglasses. They are equipped with a holographic display that generates 3D image suspended in the air,” explains Dariusz Żołna, a software engineer from MedApp, a Kraków-based company, which introduces this technology into Polish hospitals. “It is a Head-Up Display, which means that the holograms do not obscure the world around us, as it is the case with Virtual Reality headsets; instead, we see the images against our surroundings’ background. While in the case of VR the user is completely cut off from his surroundings, here he sees two realities at the same time,” he says.
But how is this possible?
“Thanks to the space recognition and user movement tracking system we can keep the hologram suspended in a specific spot in space; the user may freely move in this space and circle the hologram. He can also turn the image around and scale it up and down in size. This is handled by special SLAM algorithms, laser sensors, and cameras. Streaming to other devices, such as tablets or cell phones, over the internet is also possible,” says Żołna.
Microsoft released the first version of the HoloLens in 2016. At the time it seemed that they would be useful primarily in all sorts of design work and in video games.
“As soon as these glasses appeared on the market I immediately saw their hidden potential,” says Mateusz Kierepka, a computer scientist and the founder of MedApp. “We are actually changing reality here; we are adding something to it. After all, our organ of vision shows us only a fraction of what surrounds us. This technology allows us to overcome our handicaps and increases the scope of information we are able to see. We expand our capabilities. I have realized that we can also expand the capabilities of physicians.”
The grapevine
The company decided to write software that would enable the HoloLens to be used in medicine. The European Health Center in Otwock was the first hospital in Poland to adopt this technology. So far, the HoloLens have been used in more than 40 heart surgeries, some of which were carried out with the patient’s chest opened, while others were performed laparoscopically, i.e. not as invasively, by using a tube fitted with a telescope and a camera. The HoloLens were also used during the first surgery in Poland attended remotely by cardiologists from Belgium.
“News spreads fast; physicians talk to each other and follow the latest innovations. ‘Since they have proven their usefulness in cardiology, maybe we could use them, too?’ they ask themselves and they contact us,” says Dariusz Żołna. “And so right now we are working on solutions for oncology, orthopedics, and ophthalmology. Our system is capable of visualizing any body part: heart, lung, tooth, pelvis. In theory, the HoloLens can be used in any discipline; all that’s needed is an appropriate application.”
“I was saying as far back as 2007 that soon physicians would start prescribing applications. I was told to stop daydreaming, to put it mildly; at the time, I was called a futurist and a visionary, which were just other words for ‘madman’.”
Mateusz Kierepka, founder of MedApp
“Is this device affordable by Polish standards?” I ask.
“The new version of the HoloLens costs three and a half thousand dollars,” replies Mateusz Kierepka. “Compared to the price of many other medical devices it isn’t much. Our system is priced comparably to those of our competitors – because obviously we already have competition in Germany, America, and Japan – but it is of higher quality,” he brags. “Large companies, such as Philips or Siemens, are also working on similar solutions. The market begins to grow, but it is a good thing. Over the last three years we have developed unique techniques, so they won’t catch up with us too quickly. We have a medically-certified system that has been tested and reviewed by physicians. Our product has already been launched on the market and we are starting to sell it,” he assures me.
Futurist or madman
Kierepka is also glad that he has successfully predicted which way medicine would proceed, because when they were just starting it wasn’t that certain at all.
“I was saying as far back as 2007 that soon physicians would start prescribing applications. I was told to stop daydreaming, to put it mildly; at the time, I was called a futurist and a visionary, which were just other words for ‘madman’,” he laughs.
“At the time I worked for a company that produced refrigerating equipment. Some of those devices are also used in medicine to store drugs. It all began when a nurse at a hospital forgot to close a fridge full of vaccines. All those vaccines, which were worth as much as several expensive cars, had to be thrown away. The hospital’s director asked us then to design a special fridge dedicated for storage of drugs, with safeguards.”
Soon after that the company received another medical order, this time for an electrocardiograph.
“We equipped it with all of the latest technologies, such as connection with a cell phone or machine learning algorithms that were very advanced at the time, but would be considered very simple by today’s standards, or, to put it pompously, artificial intelligence.
”Then, the scales fell from my eyes and I realized that this is the future of medicine. I started proposing that soon everyone would be contacting their doctor by cell phone, which was seen as complete science-fiction. But I understood that mobile devices are the future, and so I started my own business. We created computer games and applications, including but not limited to simple medical applications, and in 2011 we began working on our own telemedicine system,” Kierepka reminisces.
“In 2016 I consulted our application with professor Dudek. We listened to his remarks and asked what else our application should have. I believe that communication is key, because we were creating a tool for physicians. The first guidelines that he gave us were that it would be good if the system could use a monitor to visualize in 3D the images sent to it. We did it. Professor Dudek praised us, but he wanted more. We started 3D printing, but he asked for even more. That’s when I had the idea to use the HoloLens.
“We worked on this solution all the way through 2017 and 2018. It was our primary goal, the niche that we began exploring. We presented it officially on Friday, December 9th, 2016. On Saturday another team did the same in Brazil, and on Monday – yet another in Japan. We were the pioneers.”
The revolution is here
“Using the HoloLens is easy. A surgeon can learn it in one hour. In the first version of the HoloLens, the one from 2016, you had to arrange your fingers in a specific way, but in the latest version from 2019 hand movements are fully intuitive: we can use one or both hands and three or five fingers – whatever is convenient.”
“What is it like, operating in the HoloLens?” I ask professor Dariusz Dudek.
“It is absolutely amazing,” he replies. “I frequently carry out minimally invasive procedures, without opening the patient’s chest. I operate by using catheters, most frequently entering the body through the wrist or the groin. I manipulate the catheter at whose other end is a device that I use to repair the heart.
“When I am operating by using the traditional methods of angiography and heart ultrasound, I feel that I have to transform these flat images into three dimensions in my brain. After all, I repair three-dimensional structures that are our body organs basing on two-dimensional images.
“The moment I put on the HoloLens glasses with holographic reconstruction I feel that suddenly I am in the middle of this heart. It is an incredible sensation. I am holding the catheter’s end in my hand and before my eyes I have the three-dimensional image of the heart. I am integrated – for the first time my movements are connected to what I’m seeing. I have the feeling that I am inside this heart and that I am mending it from inside.
“I believe that in the 21st century medicine accurate imaging will become increasingly necessary. It is obvious that – especially when we are not opening the patient’s chest – we should operate while using the highest quality of imaging. The old methods have become insufficient.”
“Are we going to see a revolution in medicine?”
“The revolution is already here.”
Physicians like miners
Today, it takes many years to train a surgeon to work on heart valves, to close septum defects, or to work on three-dimensional structures. The HoloLens technology seems to be the perfect educational aid for young operators.
“Today, young surgeons sometimes print the stages of the surgical procedure for themselves as an aid,” says Mateusz Kierepka. “When in doubt during the procedure, they pause it for a moment and consult with this ‘cheat sheet’ to make sure they’ve done everything as planned. When using the HoloLens, they can place several images at once in the space ahead of them: the heart before the procedure, during the procedure, as well as a simulation of how the procedure should be carried out. One look will suffice to make sure what needs to be done at any given time. A surgeon I know put it like this, ‘I spent more than a dozen years to achieve this level of surgical competence. Thanks to this technology my successors will be able to skip a few years of such training.’”
Youtube movie URL: https://www.youtube.com/watch?v=C_cmew_I-Cw
“But technology can sometimes be unreliable.”
“Obviously, a malfunction can always happen. I remember that one hospital had no power for several days and the physicians could count only on themselves. In a such a situation a surgeon, who trusts technology too much, would have a problem. On the other hand, we should remember that most physicians are very talented people with excellent memory. The HoloLens will simply allow them to complete the learning stage faster, but they will remember it later just as well as their older colleagues who trained without such tools.”
“I believe that this technology will become a natural element of surgeries,” says Zbigniew Nawrat, creator of Polish surgical robots.
“It is a matter of just a few years. Young physicians will enter the market for whom the methods that still seem experimental to us will be completely natural. At the Cardiac Surgery Development Foundation in Zabrze we are researching all methods that may aid physicians. Providing physicians with comfortable working conditions is very important. Few people know that physicians live shorter than miners; it’s a high risk profession. Work comfort is necessary so that more people decide to take up this profession and stay in it long enough. In the past, surgeons worked until they were sixty; they retired just when they obtained extensive experience and knowledge. By providing them with convenient tools for remote surgeries and diagnostics we create conditions in which they could remain professionally active much longer. Furthermore, such surgeon’s skills could be disseminated thanks to the possibility of remote transfer of information and – thanks to robots – of action. The more such innovations there are, the better.
“I am extremely glad that Mateusz has become interested in the heart and that they are doing all those fantastic things together with professor Dudek. They are pioneers; I am watching everything that is happening in this field around the world, and MedApp is in the lead.”
Machine is learning
MedApp’s medical applications feature artificial intelligence algorithms. In the case of the HoloLens artificial intelligence performs automated segmentation, which means that it selects the fragment that is of interest to the physician. Basing on the magnetic resonance imaging, computer tomography, or rotational angiography, algorithms can do it with 92 to 98% effectiveness.
AI is also used in another medical application they created. It is an application sold together with a blood pressure monitor and pulse oximeter. A set of sensors collects the data and sends them to the telemedical center. These data: temperature, blood pressure, body weight etc. are analyzed by a human, but the artificial intelligence system supervises the whole process and recommends certain actions to the professional. “If the ECG graph looks so and so and the body weight is this much, then it means that the probability of this condition is this high.” This covers both statistical methods and artificial intelligence. This way telemedical center’s employees can serve more people than usual. They has access to information already processed by the system.
“Artificial intelligence is better and faster than any human,” says Mateusz Kierepka. “Sure, sometimes it can make a mistake, because it doesn’t know whether the patient it is monitoring is not an athlete, for example. There is no personalization yet, but this feature will be implemented soon. The final decision will always be up to the physician. Limited access to data is a barrier when developing this type of technology. We have just signed an agreement with Johnson & Johnson: a dozen or so people are going to provide us with records made with such a device, but their data will be completely anonymous. The artificial intelligence will see patient no. 937, male, aged 42, but it won’t know that his name is John Smith. This way we will be able to train it better.”
Jewelry for the patient
“I remember the physicians’ reactions when we showed them our first electrocardiograph,” reminisces Kierepka. “Most were saying that it is not possible for an electrocardiograph the size of two matchboxes to produce data as reliable as a device the size of a fridge. Another problem for them was that the data were presented digitally instead of on paper. I can’t count the number of times we entered a clinic and couldn’t find a single computer… So much has changed in just a few years!”
Changes in medicine take place very rapidly.
“A few years ago I was saying that soon ECG devices would be hidden in earrings. As usual, I was treated as a futurist. First earrings with ECG functionality and a large number of sensors were made available commercially this year. This earring checks our heart rate and makes simplified ECG readouts. In ten years, doctors’ rooms will most likely have devices that are completely unknown to us today. And we may not even notice them, because they will be so small,” claims Kierepka.
“The moment I put the HoloLens glasses on, I have the feeling that I am inside this heart and that I am mending it from inside.”
professor Dariusz Dudek, cardiac surgeon
“What is the greatest problem in Poland when it comes to the development of new medical technologies?” I ask.
“Our greatest problem isn’t the lack of technology, it is environmental pollution and insufficient number of physicians. Polluted environment affects our health, including but not limited to causing problems of cardiological nature which are responsible for half of all deaths. Therefore, we need to monitor our health better. New wearable technologies continue to emerge, and their prices are getting lower. A watch that does ECG, checks if I am not suffering from arrhythmia, and alerts my physician in case of emergency cost more than two thousand zlotys last year. Today, similar watches can be bought for several hundred zlotys.”
“Will such technologies become the remedy for low numbers of physicians?”
“Technologies may enable doctors to handle more patients, and patients will be capable of managing their health better. Telemedicine will improve the availability of those physicians who do not leave Poland. It will be a blessing, especially for the people who live far from large cities.”
“What is the current condition of telemedicine in Poland?”
“The direction of changes is good, but progress is too slow. If we could increase its speed tenfold, we would become global leaders. We use digital prescriptions. The regulations are correct and even the GDPR doesn’t make such a solution more difficult to use. We just need decisions concerning payment for the services performed.
“Digital revolution has to be carried out smartly, the way Estonia did it. First, they implemented a nationwide banking system, then, basing on this system, they implemented a digital personal ID system, and only then a medical system. All those systems work together. Today, Estonia is the only country in the world that guarantees access to genetic tests for everyone. Each physician has access to them. Try to imagine what health management of such a society looks like, how much simpler it is with such a database. They are capable of testing the whole population genetically to, for example, check whether medicine X would have a beneficial or a detrimental effect on it. Basing on their experiences, we can do the exact same thing.”
Przeczytaj polską wersję tego tekstu TUTAJ