<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Cri-O on Hamzah Khan — DevOps Engineer &amp; Tech Blog</title><link>https://www.hamzahkhan.com/tags/cri-o/</link><description>Recent content in Cri-O on Hamzah Khan — DevOps Engineer &amp; Tech Blog</description><generator>Hugo</generator><language>en</language><lastBuildDate>Thu, 01 Oct 2026 14:40:00 +0000</lastBuildDate><atom:link href="https://www.hamzahkhan.com/tags/cri-o/index.xml" rel="self" type="application/rss+xml"/><item><title>Fixing Kubespray CRI-O read-only file system errors on Fedora CoreOS</title><link>https://www.hamzahkhan.com/posts/2026/2026-10-01-kubespray-cri-o-fedora-coreos-readonly-fix/</link><pubDate>Thu, 01 Oct 2026 14:40:00 +0000</pubDate><guid>https://www.hamzahkhan.com/posts/2026/2026-10-01-kubespray-cri-o-fedora-coreos-readonly-fix/</guid><description>&lt;p&gt;Deploying Kubernetes on Fedora CoreOS with Kubespray, I hit a Kubespray CRI-O read-only
file system error partway through the container runtime install. The fix is one inventory
variable.&lt;/p&gt;
&lt;p&gt;The failing task:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;TASK [container-engine/cri-o : Cri-o | create directory for libexec] ******************
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;fatal: [worker-03]: FAILED! =&amp;gt; {&amp;#34;changed&amp;#34;: false, &amp;#34;msg&amp;#34;: &amp;#34;There was an issue creating /usr/libexec/crio as requested: [Errno 30] Read-only file system: b&amp;#39;/usr/libexec/crio&amp;#39;&amp;#34;, &amp;#34;path&amp;#34;: &amp;#34;/usr/libexec/crio&amp;#34;}
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id="why-kubesprays-cri-o-role-fails-on-fedora-coreos"&gt;
Why Kubespray&amp;rsquo;s CRI-O role fails on Fedora CoreOS
&lt;a class="heading-link" href="#why-kubesprays-cri-o-role-fails-on-fedora-coreos"&gt;
&lt;i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"&gt;&lt;/i&gt;
&lt;span class="sr-only"&gt;Link to heading&lt;/span&gt;
&lt;/a&gt;
&lt;/h2&gt;
&lt;p&gt;Fedora CoreOS mounts &lt;code&gt;/usr&lt;/code&gt; read-only, so only OS updates can change it. Kubespray&amp;rsquo;s CRI-O
role sets &lt;code&gt;crio_libexec_dir&lt;/code&gt; to &lt;code&gt;/usr/libexec/crio&lt;/code&gt; by default and copies &lt;code&gt;conmon&lt;/code&gt;,
&lt;code&gt;conmonrs&lt;/code&gt;, &lt;code&gt;crun&lt;/code&gt; and &lt;code&gt;runc&lt;/code&gt; into it, so the first task that creates that directory fails.&lt;/p&gt;</description><content:encoded>&lt;p&gt;Deploying Kubernetes on Fedora CoreOS with Kubespray, I hit a Kubespray CRI-O read-only
file system error partway through the container runtime install. The fix is one inventory
variable.&lt;/p&gt;
&lt;p&gt;The failing task:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;TASK [container-engine/cri-o : Cri-o | create directory for libexec] ******************
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;fatal: [worker-03]: FAILED! =&amp;gt; {&amp;#34;changed&amp;#34;: false, &amp;#34;msg&amp;#34;: &amp;#34;There was an issue creating /usr/libexec/crio as requested: [Errno 30] Read-only file system: b&amp;#39;/usr/libexec/crio&amp;#39;&amp;#34;, &amp;#34;path&amp;#34;: &amp;#34;/usr/libexec/crio&amp;#34;}
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id="why-kubesprays-cri-o-role-fails-on-fedora-coreos"&gt;
Why Kubespray&amp;rsquo;s CRI-O role fails on Fedora CoreOS
&lt;a class="heading-link" href="#why-kubesprays-cri-o-role-fails-on-fedora-coreos"&gt;
&lt;i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"&gt;&lt;/i&gt;
&lt;span class="sr-only"&gt;Link to heading&lt;/span&gt;
&lt;/a&gt;
&lt;/h2&gt;
&lt;p&gt;Fedora CoreOS mounts &lt;code&gt;/usr&lt;/code&gt; read-only, so only OS updates can change it. Kubespray&amp;rsquo;s CRI-O
role sets &lt;code&gt;crio_libexec_dir&lt;/code&gt; to &lt;code&gt;/usr/libexec/crio&lt;/code&gt; by default and copies &lt;code&gt;conmon&lt;/code&gt;,
&lt;code&gt;conmonrs&lt;/code&gt;, &lt;code&gt;crun&lt;/code&gt; and &lt;code&gt;runc&lt;/code&gt; into it, so the first task that creates that directory fails.&lt;/p&gt;
&lt;p&gt;This comes from Kubespray&amp;rsquo;s settings for CRI-O 1.31 and later. Older versions installed
conmon into &lt;code&gt;bin_dir&lt;/code&gt; and never touched &lt;code&gt;/usr/libexec&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="fixing-the-read-only-error-with-crio_libexec_dir"&gt;
Fixing the read-only error with crio_libexec_dir
&lt;a class="heading-link" href="#fixing-the-read-only-error-with-crio_libexec_dir"&gt;
&lt;i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"&gt;&lt;/i&gt;
&lt;span class="sr-only"&gt;Link to heading&lt;/span&gt;
&lt;/a&gt;
&lt;/h2&gt;
&lt;p&gt;The path is a variable, so I override it in the inventory and point it at somewhere
writable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-yaml" data-lang="yaml"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c"&gt;# inventory/&amp;lt;cluster&amp;gt;/group_vars/all/coreos.yml&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nt"&gt;crio_libexec_dir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;/opt/libexec/crio&amp;#34;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Re-run the playbook. Kubespray installs the binaries into the new directory and writes the
matching &lt;code&gt;conmon&lt;/code&gt; and runtime paths into &lt;code&gt;/etc/crio/crio.conf&lt;/code&gt;. The &lt;code&gt;crio.service&lt;/code&gt; unit
doesn&amp;rsquo;t reference this directory, so it needs no change.&lt;/p&gt;
&lt;h2 id="checking-it-worked"&gt;
Checking it worked
&lt;a class="heading-link" href="#checking-it-worked"&gt;
&lt;i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"&gt;&lt;/i&gt;
&lt;span class="sr-only"&gt;Link to heading&lt;/span&gt;
&lt;/a&gt;
&lt;/h2&gt;
&lt;p&gt;On a node:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;grep conmon /etc/crio/crio.conf
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;systemctl is-active crio
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The &lt;code&gt;conmon&lt;/code&gt; line should point at &lt;code&gt;/opt/libexec/crio/conmon&lt;/code&gt;, and CRI-O should report
&lt;code&gt;active&lt;/code&gt;.&lt;/p&gt;</content:encoded></item></channel></rss>