[ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder

Discussion:

DJ-Pfulio via Ale

2018-01-18 13:23:16 UTC

Red Hat slams into reverse on CPU fix for Spectre design blunder

https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/

Be careful out there.

Anyone seen system instabilities from the different layers of patches?

I've delayed patching the last few weeks to let others find most of the
issues first. ;)
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Lightner, Jeffrey via Ale

2018-01-18 14:53:51 UTC

Permalink

The first box (RHEL7.4) I attempted to update panic'ed so I had to boot from the older kernel.

However, I'm not sure if that was caused by the new kernel and firmware package or the fact that I accidentally rebooted the box before it finished a full "yum update" of all packages I'd run. The interruption left it with lots of duplicate packages (the old and the updated versions) that I had to clean up. Even after the cleanup and new completed "yum update" it panic'ed on the next boot. On trying to boot from the older kernel it left the system at a place where I couldn't get into GUI console, text console or via ssh. It may be the instability RedHat is now reporting that caused the final state or it might be my fubar (or a combination).

The good news is we already had a replacement for this server planned. Rather than wasting any more time on the old server (I was able to boot into single user on old kernel) I opted to go ahead and install the replacement. It had the then latest updates but had been running fine on those. Based on RedHat's announcements I intend to run updates on it again so it will do the reversion they're recommending.

The other good news is that working with all that prevented me from being Johnny on the spot with all our other systems.

-----Original Message-----
From: Ale [mailto:ale-***@ale.org] On Behalf Of DJ-Pfulio via Ale
Sent: Thursday, January 18, 2018 8:23 AM
To: Atlanta Linux Enthusiasts
Subject: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder

Red Hat slams into reverse on CPU fix for Spectre design blunder

https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/

Be careful out there.

Anyone seen system instabilities from the different layers of patches?

I've delayed patching the last few weeks to let others find most of the issues first. ;) _______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Derek Atkins via Ale

2018-01-18 14:56:31 UTC

Permalink

My upgrades to 7.4 went smoothly on the two systems I upgraded. An
interrupted yum update could definitely cause major issues.

-derek
Sent using my mobile device. Please excuse any typos.

Post by Lightner, Jeffrey via Ale
The first box (RHEL7.4) I attempted to update panic'ed so I had to boot
from the older kernel.
However, I'm not sure if that was caused by the new kernel and firmware
package or the fact that I accidentally rebooted the box before it finished
a full "yum update" of all packages I'd run. The interruption left it
with lots of duplicate packages (the old and the updated versions) that I
had to clean up. Even after the cleanup and new completed "yum update" it
panic'ed on the next boot. On trying to boot from the older kernel it
left the system at a place where I couldn't get into GUI console, text
console or via ssh. It may be the instability RedHat is now reporting
that caused the final state or it might be my fubar (or a combination).
The good news is we already had a replacement for this server planned.
Rather than wasting any more time on the old server (I was able to boot
into single user on old kernel) I opted to go ahead and install the
replacement. It had the then latest updates but had been running fine on
those. Based on RedHat's announcements I intend to run updates on it
again so it will do the reversion they're recommending.
The other good news is that working with all that prevented me from being
Johnny on the spot with all our other systems.
-----Original Message-----
Sent: Thursday, January 18, 2018 8:23 AM
To: Atlanta Linux Enthusiasts
Subject: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;) _______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Jerald Sheets via Ale

2018-01-18 15:02:02 UTC

Permalink

I was informed by our engineers that we had relatively smooth patching, and had made it through 1/3 of our fleet before having to roll back and start applying the Intel patch. Nice thing about being in the valley is that the Intel fellow came by the offices and dropped off the patches personallyâŠ go figure.

So, we start all over patching 300k+ machines. Good times.

My upgrades to 7.4 went smoothly on the two systems I upgraded. An interrupted yum update could definitely cause major issues.
-derek
Sent using my mobile device. Please excuse any typos.

Derek Atkins via Ale

2018-01-25 01:27:16 UTC

Permalink

However, after I updated my systems Red Hat released yet another update
which apparently reverses the fix.. So if I take yet another update it
will open me up again. Feh!

-derek

Post by Jerald Sheets via Ale
I was informed by our engineers that we had relatively smooth
patching, and had made it through 1/3 of our fleet before having to
roll back and start applying the Intel patch. Nice thing about being
in the valley is that the Intel fellow came by the offices and dropped
off the patches personally… go figure.
So, we start all over patching 300k+ machines. Good times.

Post by Derek Atkins via Ale
My upgrades to 7.4 went smoothly on the two systems I upgraded. An
interrupted yum update could definitely cause major issues.
-derek
Sent using my mobile device. Please excuse any typos.

_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

--
Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
Member, MIT Student Information Processing Board (SIPB)
URL: http://web.mit.edu/warlord/ PP-ASEL-IA N1NWH
***@MIT.EDU PGP key available
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http:/

Lightner, Jeffrey via Ale

2018-01-18 15:18:16 UTC

Permalink

No doubt, but the fact I got it booted into multi-user from old kernel (before my "cleanup") made me wonder if it was really the interrupted update that caused the panic or the updated kernel and microcode combination. The fact I saw the panic even after the "cleanup" made it seem more likely it was the new kernel and microcode. The fact I couldn't get into multiuser using the old kernel after the "cleanup" I'm willing to accept was more because of the interrupted update and "cleanup".

I haven't delved into RedHat's reversal yet but the thing I'd seen even before they did the reversal suggested they already knew the microcode wasn't appropriate for all systems. Dell on the other hand wasn't offering an update for that generation of server and Intel was saying one would need to get the update from the server manufacturer. All that was why I asked a question about the catch 22 on this list before I applied anything.

-----Original Message-----
From: Derek Atkins [mailto:***@ihtfp.com]
Sent: Thursday, January 18, 2018 9:57 AM
To: Lightner, Jeffrey; DJ-Pfulio; Atlanta Linux Enthusiasts
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder

My upgrades to 7.4 went smoothly on the two systems I upgraded. An interrupted yum update could definitely cause major issues.

-derek
Sent using my mobile device. Please excuse any typos.

Post by Lightner, Jeffrey via Ale
The first box (RHEL7.4) I attempted to update panic'ed so I had to
boot from the older kernel.
However, I'm not sure if that was caused by the new kernel and
firmware package or the fact that I accidentally rebooted the box before it finished
a full "yum update" of all packages I'd run. The interruption left it
with lots of duplicate packages (the old and the updated versions) that I
had to clean up. Even after the cleanup and new completed "yum update" it
panic'ed on the next boot. On trying to boot from the older kernel it
left the system at a place where I couldn't get into GUI console, text
console or via ssh. It may be the instability RedHat is now reporting
that caused the final state or it might be my fubar (or a combination).
The good news is we already had a replacement for this server planned.
Rather than wasting any more time on the old server (I was able to
boot into single user on old kernel) I opted to go ahead and install the
replacement. It had the then latest updates but had been running fine on
those. Based on RedHat's announcements I intend to run updates on it
again so it will do the reversion they're recommending.
The other good news is that working with all that prevented me from
being Johnny on the spot with all our other systems.
-----Original Message-----
Sent: Thursday, January 18, 2018 8:23 AM
To: Atlanta Linux Enthusiasts
Subject: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of
the issues first. ;) _______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Jim Kinney via Ale

2018-01-18 15:23:34 UTC

Permalink

My $0.02

The failure is with Intel (and AMD). They should be the source of the patch. Ducking out to the vendor of the mobo is a copout. The mobo vender should already have tools for board level firmware upgrades. This vulnerability doesn't, as far as I can tell, impact a firmware upgrade. If it does, that's a new problem.

Post by Lightner, Jeffrey via Ale
No doubt, but the fact I got it booted into multi-user from old kernel
(before my "cleanup") made me wonder if it was really the interrupted
update that caused the panic or the updated kernel and microcode
combination. The fact I saw the panic even after the "cleanup" made
it seem more likely it was the new kernel and microcode. The fact I
couldn't get into multiuser using the old kernel after the "cleanup"
I'm willing to accept was more because of the interrupted update and
"cleanup".
I haven't delved into RedHat's reversal yet but the thing I'd seen even
before they did the reversal suggested they already knew the microcode
wasn't appropriate for all systems. Dell on the other hand wasn't
offering an update for that generation of server and Intel was saying
one would need to get the update from the server manufacturer. All
that was why I asked a question about the catch 22 on this list before
I applied anything.
-----Original Message-----
Sent: Thursday, January 18, 2018 9:57 AM
To: Lightner, Jeffrey; DJ-Pfulio; Atlanta Linux Enthusiasts
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder
My upgrades to 7.4 went smoothly on the two systems I upgraded. An
interrupted yum update could definitely cause major issues.
-derek
Sent using my mobile device. Please excuse any typos.
On January 18, 2018 9:54:18 AM "Lightner, Jeffrey via Ale"

before it finished

Post by Lightner, Jeffrey via Ale
a full "yum update" of all packages I'd run. The interruption left

Post by Lightner, Jeffrey via Ale
with lots of duplicate packages (the old and the updated versions)

that I

Post by Lightner, Jeffrey via Ale
had to clean up. Even after the cleanup and new completed "yum

update" it

Post by Lightner, Jeffrey via Ale
panic'ed on the next boot. On trying to boot from the older kernel

Post by Lightner, Jeffrey via Ale
left the system at a place where I couldn't get into GUI console,

text

Post by Lightner, Jeffrey via Ale
console or via ssh. It may be the instability RedHat is now

reporting

Post by Lightner, Jeffrey via Ale
that caused the final state or it might be my fubar (or a

combination).

Post by Lightner, Jeffrey via Ale
The good news is we already had a replacement for this server

planned.

Post by Lightner, Jeffrey via Ale
Rather than wasting any more time on the old server (I was able to
boot into single user on old kernel) I opted to go ahead and install

the

Post by Lightner, Jeffrey via Ale
replacement. It had the then latest updates but had been running

fine on

Post by Lightner, Jeffrey via Ale
those. Based on RedHat's announcements I intend to run updates on

Post by Lightner, Jeffrey via Ale
again so it will do the reversion they're recommending.
The other good news is that working with all that prevented me from
being Johnny on the spot with all our other systems.
-----Original Message-----
Sent: Thursday, January 18, 2018 8:23 AM
To: Atlanta Linux Enthusiasts
Subject: [ale] News: Red Hat slams into reverse on CPU fix for

Spectre

Post by Lightner, Jeffrey via Ale
design blunder
Red Hat slams into reverse on CPU fix for Spectre design blunder

https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda

Post by Lightner, Jeffrey via Ale
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of

patches?

Post by Lightner, Jeffrey via Ale
I've delayed patching the last few weeks to let others find most of
the issues first. ;) _______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

--
Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.

Lightner, Jeffrey via Ale

2018-01-18 15:39:33 UTC

Permalink

I agree.

Youâd think Intel would learn from the Solaris CPU debacle a few years back or even their own initial Pentium issues previously that disclaiming responsibility and not providing a fix does not succeed.

From: Jim Kinney [mailto:***@gmail.com]
Sent: Thursday, January 18, 2018 10:24 AM
To: Lightner, Jeffrey; Atlanta Linux Enthusiasts; Lightner, Jeffrey via Ale; Derek Atkins; DJ-Pfulio
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder

My $0.02

The failure is with Intel (and AMD). They should be the source of the patch. Ducking out to the vendor of the mobo is a copout. The mobo vender should already have tools for board level firmware upgrades. This vulnerability doesn't, as far as I can tell, impact a firmware upgrade. If it does, that's a new problem.

Ted W. via Ale

2018-01-22 03:13:09 UTC

Permalink

Thanks for the heads up on this. So far I've done a handful of Redhat
kernel upgrades on 7.3 and haven't had any issues. The only problems
I've had so far are my clusters running 4.4 kernels from elrepo. The
4.4.110 kernel works fine but, as predicted, the Intel 10GbE module
built in to the packaged initramfs is broken (still) and I'm having to
compile my own module from Intel sources and strip the bad one from
initramfs. No related issues WRT spectre/meltdown patches, though.
Though, I do wish the upstream kernel had the flags Redhat built in to
their kernel so I could get some more concrete benchmark data out of it.

Post by Lightner, Jeffrey via Ale
The first box (RHEL7.4) I attempted to update panic'ed so I had to boot from the older kernel.
However, I'm not sure if that was caused by the new kernel and firmware package or the fact that I accidentally rebooted the box before it finished a full "yum update" of all packages I'd run. The interruption left it with lots of duplicate packages (the old and the updated versions) that I had to clean up. Even after the cleanup and new completed "yum update" it panic'ed on the next boot. On trying to boot from the older kernel it left the system at a place where I couldn't get into GUI console, text console or via ssh. It may be the instability RedHat is now reporting that caused the final state or it might be my fubar (or a combination).
The good news is we already had a replacement for this server planned. Rather than wasting any more time on the old server (I was able to boot into single user on old kernel) I opted to go ahead and install the replacement. It had the then latest updates but had been running fine on those. Based on RedHat's announcements I intend to run updates on it again so it will do the reversion they're recommending.
The other good news is that working with all that prevented me from being Johnny on the spot with all our other systems.
-----Original Message-----
Sent: Thursday, January 18, 2018 8:23 AM
To: Atlanta Linux Enthusiasts
Subject: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the issues first. ;) _______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Beddingfield, Allen via Ale

2018-01-18 16:12:50 UTC

Permalink

So, my question at this point:
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .

Allen B.

Post by DJ-Pfulio via Ale
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Jim Kinney via Ale

2018-01-18 16:16:06 UTC

Permalink

Please check with other distros! RedHat is probably taking a lead in building a patch as they have resources. I've seen other distros issue kernel patches after rhel that are rpm based.

Post by Beddingfield, Allen via Ale
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.

Post by DJ-Pfulio via Ale
Red Hat slams into reverse on CPU fix for Spectre design blunder

https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/

Post by DJ-Pfulio via Ale
Be careful out there.
Anyone seen system instabilities from the different layers of

patches?

Post by DJ-Pfulio via Ale
I've delayed patching the last few weeks to let others find most of

the

Post by DJ-Pfulio via Ale
issues first. ;)
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

--
Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.

Beddingfield, Allen via Ale

2018-01-18 16:18:16 UTC

Permalink

I'm going to throw the question out to some SUSE folks and see what
answer I get. They would be the other one with the resources for
independent testing . . .

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_update_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)
------------------------------------------------------------------------
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
Sent from my Android device with K-9 Mail. All tyopes are thumb related
and reflect authenticity.

Lightner, Jeffrey via Ale

2018-01-18 16:21:55 UTC

Permalink

RHEL was issuing both a kernel patch and a microcode update for CPUs. It may be other distros weren't doing the microcode update. In RHEL's reversal they say they did the microcode as a convenience but view it as the chip maker's responsibility. The reversal was to get rid of the RHEL microcode update and take out optimizations to the kernel that relied upon it.

Maybe SUSE and others weren't also providing a microcode update.

Note that without a CPU firmware/microcode update you're not protected against Spectre but may be protected against Meltdown.

-----Original Message-----
From: Ale [mailto:ale-***@ale.org] On Behalf Of Beddingfield, Allen via Ale
Sent: Thursday, January 18, 2018 11:18 AM
To: Jim Kinney; Atlanta Linux Enthusiasts
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder

I'm going to throw the question out to some SUSE folks and see what answer I get. They would be the other one with the resources for independent testing . . .

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)
----------------------------------------------------------------------
--
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
Sent from my Android device with K-9 Mail. All tyopes are thumb
related and reflect authenticity.

--
Allen Beddingfield
Systems Engineer
Office of Information Technology
The University of Alabama
Office 205-348-2251
***@ua.edu
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Beddingfield, Allen via Ale

2018-01-18 16:23:42 UTC

Permalink

SUSE is providing the microcode. I can confirm that.
So far, they don't seem to have pulled it.
Allen B.

Post by Lightner, Jeffrey via Ale
RHEL was issuing both a kernel patch and a microcode update for CPUs. It may be other distros weren't doing the microcode update. In RHEL's reversal they say they did the microcode as a convenience but view it as the chip maker's responsibility. The reversal was to get rid of the RHEL microcode update and take out optimizations to the kernel that relied upon it.
Maybe SUSE and others weren't also providing a microcode update.
Note that without a CPU firmware/microcode update you're not protected against Spectre but may be protected against Meltdown.
-----Original Message-----
Sent: Thursday, January 18, 2018 11:18 AM
To: Jim Kinney; Atlanta Linux Enthusiasts
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder
I'm going to throw the question out to some SUSE folks and see what answer I get. They would be the other one with the resources for independent testing . . .

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)
----------------------------------------------------------------------
--
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
Sent from my Android device with K-9 Mail. All tyopes are thumb
related and reflect authenticity.

--
Allen Beddingfield
Systems Engineer
Office of Information Technology
The University of Alabama
Office 205-348-2251
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

James Taylor via Ale

2018-01-18 19:41:31 UTC

Permalink

I've applied to to a couple of my servers as a test, and there was no drama.
-jt

James Taylor
678-697-9420
SUSE is providing the microcode. I can confirm that.
So far, they don't seem to have pulled it.
Allen B.

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)
----------------------------------------------------------------------
--
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
Sent from my Android device with K-9 Mail. All tyopes are thumb
related and reflect authenticity.

--
Allen Beddingfield
Systems Engineer
Office of Information Technology
The University of Alabama
Office 205-348-2251
***@ua.edu
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Beddingfield, Allen via Ale

2018-01-18 20:13:22 UTC

Permalink

I've rolled the SLES 12 SP3 fix out to our production web servers. No
issues so far...

Post by James Taylor via Ale
I've applied to to a couple of my servers as a test, and there was no drama.
-jt
James Taylor
678-697-9420
SUSE is providing the microcode. I can confirm that.
So far, they don't seem to have pulled it.
Allen B.

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of
patches?
I've delayed patching the last few weeks to let others find most
of the
issues first. ;)
----------------------------------------------------------------------
--
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
Sent from my Android device with K-9 Mail. All tyopes are thumb
related and reflect authenticity.

DJ-Pfulio via Ale

2018-01-18 16:29:06 UTC

Permalink

Ubuntu has a microcode package for intel (intel-microcode) or amd CPUs.
I don't know if that package is included by default or not. It was
already installed on my 16.04 laptop and a 14.04.5 server, so when I do
patch, the updated microcode will come to those systems.

Also, x32 Ubuntu isn't high priority, so people running non-64-bit
installs shouldn't expect patches until after all the x64 stuff is solved.

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Horkan Smith via Ale

2018-01-18 17:24:15 UTC

Permalink

TL;DR - some people had problems w/ Ubuntu desktop, others were ok; Intel microcode may be available for Ubuntu but not default.

FWIW (incomplete, hearsay evidence so not much);

We mostly use Ubuntu desktop and derivatives in house (small shop w/ 6-7 active linux users); I heard from 3 that experienced 'issues' after the update - two had multiple apparent freezes-requiring-reboot, one had multiple machines that would boot, but be very slow or unable to open firefox, chromium, or even gnome-terminal & would fail w/ a compiz crash. Also FWIW, at least two had the 4.13.0-26-generic kernel (Ubuntu 16.04.3 release, 16.04.1 seems to be at 4.4.0-109-generic), and I think they both use the Intel graphics driver.

On the other hand, I've been running the upgrade on both .1 and .3 w/ out issues on 2 Nvidia and 1 AMD machine. (The AMD may not be good evidence, it's mostly used headless.) I also tend to use a lightweight DE (LXDE or fvwm).

Also, I'm pretty sure there's an Intel microcode that's considered proprietary, which means you have to explicitly check the 3rd party box to enable it, then install it. You might also have to turn off secure boot - I know you do for the Nvidia drivers. I'm not positive if that's the same microcode they patch, however.

later!
horkan

Post by DJ-Pfulio via Ale
Ubuntu has a microcode package for intel (intel-microcode) or amd CPUs.
I don't know if that package is included by default or not. It was
already installed on my 16.04 laptop and a 14.04.5 server, so when I do
patch, the updated microcode will come to those systems.
Also, x32 Ubuntu isn't high priority, so people running non-64-bit
installs shouldn't expect patches until after all the x64 stuff is solved.

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)

_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

--
Horkan Smith
678-777-3263 cell, ***@horkan.net
_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Derek Atkins via Ale

2018-01-18 16:30:14 UTC

Permalink

It looks like RHEL/CentOS just released yet another microcode/firmware
update for 7.4 yesterday. The previous one from from about a week or so
ago (Jan 4th).

-derek

Post by Lightner, Jeffrey via Ale
RHEL was issuing both a kernel patch and a microcode update for CPUs. It
may be other distros weren't doing the microcode update. In RHEL's
reversal they say they did the microcode as a convenience but view it as
the chip maker's responsibility. The reversal was to get rid of the RHEL
microcode update and take out optimizations to the kernel that relied upon
it.
Maybe SUSE and others weren't also providing a microcode update.
Note that without a CPU firmware/microcode update you're not protected
against Spectre but may be protected against Meltdown.
-----Original Message-----
Sent: Thursday, January 18, 2018 11:18 AM
To: Jim Kinney; Atlanta Linux Enthusiasts
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder
I'm going to throw the question out to some SUSE folks and see what answer
I get. They would be the other one with the resources for independent
testing . . .

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros issue
kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upda
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)
----------------------------------------------------------------------
--
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
Sent from my Android device with K-9 Mail. All tyopes are thumb
related and reflect authenticity.

--
Derek Atkins 617-623-3745
***@ihtfp.com www.ihtfp.com
Computer and Internet Security Consultant

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Jerald Sheets via Ale

2018-01-18 17:11:18 UTC

Permalink

Hot on the heels of this conversation, we have this:

https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr <https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr>

âj

Lightner, Jeffrey via Ale

2018-01-18 19:20:18 UTC

Permalink

What RHEL (and therefore CentOS) released yesterday reverts to the microcode version before the one they released a week or so ago trying to fix Spectre. That is to say if you install/update the "patch" you essentially are uninstalling/downgrading the microcode version.

After the microcode downgrade you'd need to get a hardware vendor supplied packages to address Spectre. Ideally this would come from Intel or AMD but if you are using a 3rd party manufacturer (e.g. Dell, Lenovo, HP) for your system that includes one of the affected processors you would need to get the update from that 3rd party (e.g. a BIOS/Firmware update).

Dell when I checked a week or so ago was only providing an update for the last 2 generations of servers so any older system is in a catch 22 as the Distro thinks the hardware provider should provide the patch, the chip makers think the folks that built the systems around their chips should provide it to customers who bought those systems and the folks that built the systems feel they have no obligation to provide such updates for otherwise unsupported systems.

Given that this flaw is said to exist in most chips built in the last 10 years I think the chip makers should be the ones providing the updates to end users regardless of whether the systems built around the systems are supported by their makers or not. I'm sure the lawyers are going to have a field day with refusal to do so.

-----Original Message-----
From: Derek Atkins [mailto:***@ihtfp.com]
Sent: Thursday, January 18, 2018 11:30 AM
To: Lightner, Jeffrey; Atlanta Linux Enthusiasts
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder

It looks like RHEL/CentOS just released yet another microcode/firmware update for 7.4 yesterday. The previous one from about a week or so ago (Jan 4th).

-derek

Post by Lightner, Jeffrey via Ale
RHEL was issuing both a kernel patch and a microcode update for CPUs. It
may be other distros weren't doing the microcode update. In RHEL's
reversal they say they did the microcode as a convenience but view it as
the chip maker's responsibility. The reversal was to get rid of the RHEL
microcode update and take out optimizations to the kernel that relied
upon it.
Maybe SUSE and others weren't also providing a microcode update.
Note that without a CPU firmware/microcode update you're not protected
against Spectre but may be protected against Meltdown.
-----Original Message-----
Sent: Thursday, January 18, 2018 11:18 AM
To: Jim Kinney; Atlanta Linux Enthusiasts
Subject: Re: [ale] News: Red Hat slams into reverse on CPU fix for Spectre design blunder
I'm going to throw the question out to some SUSE folks and see what
answer I get. They would be the other one with the resources for
independent testing . . .

Post by Jim Kinney via Ale
Please check with other distros! RedHat is probably taking a lead in
building a patch as they have resources. I've seen other distros
issue kernel patches after rhel that are rpm based.
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good" if
we have SUSE and Debian/Ubuntu systems patched up, and that it is just
the RHEL world that can't get their act together? It seems to me that
we are likely to hear a "me too" out of the other vendors
following Red
Hat's announcement. Thoughts? I'm going to pose the same question to a
SUSE heavy audience, see what I get, and report back . . .
Allen B.
Red Hat slams into reverse on CPU fix for Spectre design blunder
https://www.theregister.co.uk/2018/01/18/red_hat_spectre_firmware_upd
a
te_woes/
Be careful out there.
Anyone seen system instabilities from the different layers of patches?
I've delayed patching the last few weeks to let others find most of the
issues first. ;)
---------------------------------------------------------------------
-
--
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo
--
Sent from my Android device with K-9 Mail. All tyopes are thumb
related and reflect authenticity.

DJ-Pfulio via Ale

2018-01-18 16:17:36 UTC

Permalink

Ubuntu:
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
Bookmark that page. They have been updating it.

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http

Beddingfield, Allen via Ale

2018-01-18 16:21:33 UTC

Permalink

This is SUSE's equivalent page, where they still say they are shipping
the microcode with the patch.

https://www.suse.com/support/kb/doc/?id=7022512

Post by DJ-Pfulio via Ale
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
Bookmark that page. They have been updating it.

_______________________________________________
Ale mailing list
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo

Jim Lynch via Ale

2018-01-18 20:23:15 UTC

Permalink

I seem to recall that Ubuntu servers were hanging at boot time after the
most recent updates which supposedly addressed the current Intel issues.

Jim.

Post by Beddingfield, Allen via Ale
We haven't heard anything similar out of SUSE, and I haven't heard of
anything in the Debian/Ubuntu world? Are we to assume we are "good"
if we have SUSE and Debian/Ubuntu systems patched up, and that it is
just the RHEL world that can't get their act together? It seems to me
that we are likely to hear a "me too" out of the other vendors
following Red Hat's announcement. Thoughts? I'm going to pose the
same question to a SUSE heavy audience, see what I get, and report
back . . .
Allen B.

_______________________________________________
Ale mailing list
***@ale.org
http://mail.ale.org/mailman/listinfo/ale
See JOBS, ANNOUNCE and SCHOOLS lists at
http://mail.ale.org/mailman/listinfo