Discussion:
[PATCH] Implement SM3 hash algorithm in gnulib and the computing tool in coreutils.
Add Reply
Pádraig Brady
2017-10-11 07:15:07 UTC
Reply
Permalink
Raw Message
Hi,
This is the review request for SM3 hash algorithm. SM3 hash algorithm is
already
accepted and supported by TPM 2.0 spec. See page 284 in TPM 2.0 spec part 1
(https://trustedcomputinggroup.org/wp-content/uploads/TPM-Rev-2.0-Part-1-Ar
chitecture-01.38.pdf)
Currently, I only implement SM3 in gnulib. I will implement it in
libgcrypt and
openssl later.
Plz refer to the following PRs for code review.
- For the new gnulib module crypto/sm3
https://github.com/coreutils/gnulib/pull/3/commits
- For the new program sm3sum
https://github.com/coreutils/coreutils/pull/12/commits
I see the GMssl fork of openssl on github.
Has any attempt been made to merge with that project?
I'm not sure how generally useful a sm3sum command would be TBH.
Being available in gnulib is more plausible,
though I'd like to see it in the more specific openssl library first.
Why the specific need for an sm3sum command?
BTW you'd need to assign copyright for this to be merged to gnulib/coreutils.

thanks for taking the time to provide patches.

Pádraig
Bruno Haible
2017-10-11 09:13:25 UTC
Reply
Permalink
Raw Message
Post by Pádraig Brady
Currently, I only implement SM3 in gnulib. I will implement it in
libgcrypt and
openssl later.
Yes, for a crypto algorjthm to be widely usable in GNU, it needs to be
available both in GNUTLS (via libgcrypt) and openssl.
Post by Pádraig Brady
Being available in gnulib is more plausible,
though I'd like to see it in the more specific openssl library first.
Indeed, we don't have many crypto experts on the gnulib mailing lists.
Therefore, I too would like the crypto experts on the libgcrypt or openssl
mailing lists to evaluate the SM3 algorithm and your code first, before we
can accept it in gnulib.

Bruno
张佳(乾越)
2017-10-11 09:16:41 UTC
Reply
Permalink
Raw Message
Post by Bruno Haible
Post by Pádraig Brady
Currently, I only implement SM3 in gnulib. I will implement it in
libgcrypt and
openssl later.
Yes, for a crypto algorjthm to be widely usable in GNU, it needs to be
available both in GNUTLS (via libgcrypt) and openssl.
Post by Pádraig Brady
Being available in gnulib is more plausible,
though I'd like to see it in the more specific openssl library first.
Indeed, we don't have many crypto experts on the gnulib mailing lists.
Therefore, I too would like the crypto experts on the libgcrypt or openssl
mailing lists to evaluate the SM3 algorithm and your code first, before we
can accept it in gnulib.
This sounds like the prerequisite of merging crypto/sm3 and sm3sum right?

Thanks,
Jia
Post by Bruno Haible
Bruno
Jia Zhang
2017-10-24 08:36:49 UTC
Reply
Permalink
Raw Message
Post by Bruno Haible
Currently, I only implement SM3 in gnulib. I will implement it
in libgcrypt and openssl later.
Yes, for a crypto algorjthm to be widely usable in GNU, it needs
to be available both in GNUTLS (via libgcrypt) and openssl.
Being available in gnulib is more plausible, though I'd like to
see it in the more specific openssl library first.
Indeed, we don't have many crypto experts on the gnulib mailing
lists. Therefore, I too would like the crypto experts on the
libgcrypt or openssl mailing lists to evaluate the SM3 algorithm
and your code first, before we can accept it in gnulib.
libgcrypt has merged SM3 hash function. See
https://dev.gnupg.org/rC4423bf3cc4432b9bfe801ff74cb05e6f0dd3eccd

I plan to add addition commits to use SM3 functions exported by
libgcrypt, and current implementation is used as baseline for review.
Is it good for you?

Thanks,
Jia
Post by Bruno Haible
Bruno
Bruno Haible
2017-10-24 10:09:56 UTC
Reply
Permalink
Raw Message
Hi Jia,
Post by Jia Zhang
Post by Bruno Haible
Indeed, we don't have many crypto experts on the gnulib mailing
lists. Therefore, I too would like the crypto experts on the
libgcrypt or openssl mailing lists to evaluate the SM3 algorithm
and your code first, before we can accept it in gnulib.
libgcrypt has merged SM3 hash function. See
https://dev.gnupg.org/rC4423bf3cc4432b9bfe801ff74cb05e6f0dd3eccd
Or [1].

Thanks. I see it wasn't an easy discussion [2], but now it's over,
so I'm awaiting your gnulib patches.
Post by Jia Zhang
I plan to add addition commits to use SM3 functions exported by
libgcrypt, and current implementation is used as baseline for review.
Yes, good.

Bruno

[1] https://git.gnupg.org/cgi-bin/gitweb.cgi?p=libgcrypt.git;a=shortlog;h=refs/heads/master
[2] https://lists.gnupg.org/pipermail/gcrypt-devel/2017-October/thread.html
Assaf Gordon
2017-10-29 07:28:48 UTC
Reply
Permalink
Raw Message
Hello,
Post by Jia Zhang
Post by Bruno Haible
Indeed, we don't have many crypto experts on the gnulib mailing
lists. Therefore, I too would like the crypto experts on the
libgcrypt or openssl mailing lists to evaluate the SM3 algorithm
and your code first, before we can accept it in gnulib.
libgcrypt has merged SM3 hash function. See
https://dev.gnupg.org/rC4423bf3cc4432b9bfe801ff74cb05e6f0dd3eccd
I see SM3 has been added to gnulib and libgcrypt,
but I would suggest we wait for further evaluation before adding it to
coreutils.

libgcrypt specifically say: "Thorough understanding of
applied cryptography is required to use Libgcrypt."
and Werner Koch explicitly acknowledged that they are willing to include
known weak algorithms in libgcrypt because non-experts should never use
it directly:
https://lists.gnupg.org/pipermail/gcrypt-devel/2017-October/004287.html

I'm assuming the same applies to gnulib - but not to coreutils.




I won't claim to have any understanding of cryptography or hashing,
but a cursory look at the code raises three red flags:

First,
It is *very* similar to sha256.
Not just the function structure, but almost line-for-line identical.
It could almost be said that SM3 is just a modified sha256 with slightly
different primitives.
(try "diff -u lib/sm3.c lib/sha256.c" to see what I mean).

sha256 was published in 2001, and while it has not yet been compromised,
I would naively think it is not wise to offer a new hashing algorithm in
2017 that's based on sha256.
I think it is already generally recommended to move away from sha-256
to at least sha-512 (or sha-3).



Second,
All SHA-2 hash functions use initialization values that
are based on the cube roots of the first 62 primes.
In lib/sha256.c it is the "sha256_round_constants" variable.
In lib/sm3.c these initialization values are the same 32bit values,
rotated:

/* SM3 round constants */
#define T(j) sm3_round_constants[j]
static const uint32_t sm3_round_constants[64] = {
0x79cc4519UL, 0xf3988a32UL, 0xe7311465UL, 0xce6228cbUL,
0x9cc45197UL, 0x3988a32fUL, 0x7311465eUL, 0xe6228cbcUL,
0xcc451979UL, 0x988a32f3UL, 0x311465e7UL, 0x6228cbceUL,
0xc451979cUL, 0x88a32f39UL, 0x11465e73UL, 0x228cbce6UL,
0x9d8a7a87UL, 0x3b14f50fUL, 0x7629ea1eUL, 0xec53d43cUL,
0xd8a7a879UL, 0xb14f50f3UL, 0x629ea1e7UL, 0xc53d43ceUL,
0x8a7a879dUL, 0x14f50f3bUL, 0x29ea1e76UL, 0x53d43cecUL,
0xa7a879d8UL, 0x4f50f3b1UL, 0x9ea1e762UL, 0x3d43cec5UL,
0x7a879d8aUL, 0xf50f3b14UL, 0xea1e7629UL, 0xd43cec53UL,
0xa879d8a7UL, 0x50f3b14fUL, 0xa1e7629eUL, 0x43cec53dUL,
0x879d8a7aUL, 0x0f3b14f5UL, 0x1e7629eaUL, 0x3cec53d4UL,
0x79d8a7a8UL, 0xf3b14f50UL, 0xe7629ea1UL, 0xcec53d43UL,
0x9d8a7a87UL, 0x3b14f50fUL, 0x7629ea1eUL, 0xec53d43cUL,
0xd8a7a879UL, 0xb14f50f3UL, 0x629ea1e7UL, 0xc53d43ceUL,
0x8a7a879dUL, 0x14f50f3bUL, 0x29ea1e76UL, 0x53d43cecUL,
0xa7a879d8UL, 0x4f50f3b1UL, 0x9ea1e762UL, 0x3d43cec5UL,




Third,
The 64 rounds ("Compression function") in sha256 seem to
rotate the internal state variables equally (a..h)
in the R macro definition [1] and usage [2].

[1] https://opengrok.housegordon.com/source/xref/gnulib/lib/sha256.c#484
[2] https://opengrok.housegordon.com/source/xref/gnulib/lib/sha256.c#504

In sm3, it seems from a cursory look that the first 128 bits
and the last 128 bits of internal states are kept mostly separated
(variable a..d and e..h) based on the R macro [3] and usage [4].

[3] https://opengrok.housegordon.com/source/xref/gnulib/lib/sm3.c#377
[4] https://opengrok.housegordon.com/source/xref/gnulib/lib/sm3.c#413



Again, I'm not a cryptography expert, and perhaps this is secure.
But it does look a bit suspicious.


---

It's one thing to add it to a library, where it can not be accidentally
used by non-expert users, but it's a different thing to provide an
executable that will be shipped by default on all future GNU/Linux
machine.

I would humbly suggest we wait until other crypto experts evaluate
'sm3', and decide to offer it in their programs (preferably openssl /
libressl) as a general-purpose program.





regards,
- assaf
Pádraig Brady
2017-10-29 21:29:15 UTC
Reply
Permalink
Raw Message
Post by Assaf Gordon
Hello,
Post by Jia Zhang
Post by Bruno Haible
Indeed, we don't have many crypto experts on the gnulib mailing
lists. Therefore, I too would like the crypto experts on the
libgcrypt or openssl mailing lists to evaluate the SM3 algorithm
and your code first, before we can accept it in gnulib.
libgcrypt has merged SM3 hash function. See
https://dev.gnupg.org/rC4423bf3cc4432b9bfe801ff74cb05e6f0dd3eccd
I see SM3 has been added to gnulib and libgcrypt,
but I would suggest we wait for further evaluation before adding it to
coreutils.
libgcrypt specifically say: "Thorough understanding of
applied cryptography is required to use Libgcrypt."
and Werner Koch explicitly acknowledged that they are willing to include
known weak algorithms in libgcrypt because non-experts should never use
https://lists.gnupg.org/pipermail/gcrypt-devel/2017-October/004287.html
I'm assuming the same applies to gnulib - but not to coreutils.
I won't claim to have any understanding of cryptography or hashing,
First,
It is *very* similar to sha256.
Not just the function structure, but almost line-for-line identical.
It could almost be said that SM3 is just a modified sha256 with slightly
different primitives.
(try "diff -u lib/sm3.c lib/sha256.c" to see what I mean).
sha256 was published in 2001, and while it has not yet been compromised,
I would naively think it is not wise to offer a new hashing algorithm in
2017 that's based on sha256.
I think it is already generally recommended to move away from sha-256
to at least sha-512 (or sha-3).
Second,
All SHA-2 hash functions use initialization values that
are based on the cube roots of the first 62 primes.
In lib/sha256.c it is the "sha256_round_constants" variable.
In lib/sm3.c these initialization values are the same 32bit values,
/* SM3 round constants */
#define T(j) sm3_round_constants[j]
static const uint32_t sm3_round_constants[64] = {
0x79cc4519UL, 0xf3988a32UL, 0xe7311465UL, 0xce6228cbUL,
0x9cc45197UL, 0x3988a32fUL, 0x7311465eUL, 0xe6228cbcUL,
0xcc451979UL, 0x988a32f3UL, 0x311465e7UL, 0x6228cbceUL,
0xc451979cUL, 0x88a32f39UL, 0x11465e73UL, 0x228cbce6UL,
0x9d8a7a87UL, 0x3b14f50fUL, 0x7629ea1eUL, 0xec53d43cUL,
0xd8a7a879UL, 0xb14f50f3UL, 0x629ea1e7UL, 0xc53d43ceUL,
0x8a7a879dUL, 0x14f50f3bUL, 0x29ea1e76UL, 0x53d43cecUL,
0xa7a879d8UL, 0x4f50f3b1UL, 0x9ea1e762UL, 0x3d43cec5UL,
0x7a879d8aUL, 0xf50f3b14UL, 0xea1e7629UL, 0xd43cec53UL,
0xa879d8a7UL, 0x50f3b14fUL, 0xa1e7629eUL, 0x43cec53dUL,
0x879d8a7aUL, 0x0f3b14f5UL, 0x1e7629eaUL, 0x3cec53d4UL,
0x79d8a7a8UL, 0xf3b14f50UL, 0xe7629ea1UL, 0xcec53d43UL,
0x9d8a7a87UL, 0x3b14f50fUL, 0x7629ea1eUL, 0xec53d43cUL,
0xd8a7a879UL, 0xb14f50f3UL, 0x629ea1e7UL, 0xc53d43ceUL,
0x8a7a879dUL, 0x14f50f3bUL, 0x29ea1e76UL, 0x53d43cecUL,
0xa7a879d8UL, 0x4f50f3b1UL, 0x9ea1e762UL, 0x3d43cec5UL,
Third,
The 64 rounds ("Compression function") in sha256 seem to
rotate the internal state variables equally (a..h)
in the R macro definition [1] and usage [2].
[1] https://opengrok.housegordon.com/source/xref/gnulib/lib/sha256.c#484
[2] https://opengrok.housegordon.com/source/xref/gnulib/lib/sha256.c#504
In sm3, it seems from a cursory look that the first 128 bits
and the last 128 bits of internal states are kept mostly separated
(variable a..d and e..h) based on the R macro [3] and usage [4].
[3] https://opengrok.housegordon.com/source/xref/gnulib/lib/sm3.c#377
[4] https://opengrok.housegordon.com/source/xref/gnulib/lib/sm3.c#413
Again, I'm not a cryptography expert, and perhaps this is secure.
But it does look a bit suspicious.
---
It's one thing to add it to a library, where it can not be accidentally
used by non-expert users, but it's a different thing to provide an
executable that will be shipped by default on all future GNU/Linux
machine.
I would humbly suggest we wait until other crypto experts evaluate
'sm3', and decide to offer it in their programs (preferably openssl /
libressl) as a general-purpose program.
Thanks for looking at this Assaf.
I do agree that a general command released to everyone
has a higher bar than routines being available in a library.
As I mentioned previously inclusion in openssl should be
a minimum prerequisite, and even then making sm3sum generally
available should be carefully considered.

cheers,
Pádraig

张佳(乾越)
2017-10-11 08:05:30 UTC
Reply
Permalink
Raw Message
Post by Pádraig Brady
Hi,
This is the review request for SM3 hash algorithm. SM3 hash algorithm is
already
accepted and supported by TPM 2.0 spec. See page 284 in TPM 2.0 spec part 1
(https://trustedcomputinggroup.org/wp-content/uploads/TPM-Rev-2.0-Part-1-Ar
chitecture-01.38.pdf)
Currently, I only implement SM3 in gnulib. I will implement it in
libgcrypt and
openssl later.
Plz refer to the following PRs for code review.
- For the new gnulib module crypto/sm3
https://github.com/coreutils/gnulib/pull/3/commits
- For the new program sm3sum
https://github.com/coreutils/coreutils/pull/12/commits
I see the GMssl fork of openssl on github.
Has any attempt been made to merge with that project?
I don't see GMssl keeps syncing up with the latest openssl so it is not
my goal.
Post by Pádraig Brady
I'm not sure how generally useful a sm3sum command would be TBH.
Being available in gnulib is more plausible,
though I'd like to see it in the more specific openssl library first.
If this is the prerequisite to merge sm3sum and crypto/sm3, I'll do it
later. But technically it is fine to compile sm3sum program without
openssl.
Post by Pádraig Brady
Why the specific need for an sm3sum command?
In some areas, SM3 algorithm is widely used, because SM3 algorithm has
additional strengthening features against sha256 at the cose of small
overhead increased.
Post by Pádraig Brady
BTW you'd need to assign copyright for this to be merged to gnulib/coreutils.
No problem. Is there anything to do for this?

Thanks,
Jia
Post by Pádraig Brady
thanks for taking the time to provide patches.
Pádraig
Loading...