usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

R

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove my middle name.

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Remove $FreeBSD$: one-line .c comment pattern

Remove /^/[*/]\s*\$FreeBSD\$.*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

usr.bin/grep: Fix Address OOB read error

I found this when compiling all the bootstrap tools with -fsanitize=addres:

==65590==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d000008400

usr.bin/grep: Fix Address OOB read error

I found this when compiling all the bootstrap tools with -fsanitize=addres:

==65590==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d000008400 at pc 0x000000473053 bp 0x7ffc1c7dd910 sp 0x7ffc1c7dd0b8
READ of size 32769 at 0x62d000008400 thread T0
#0 0x473052 in regexec (/local/scratch/alr48/cheri/build/freebsd-amd64-build/local/scratch/alr48/cheri/freebsd/amd64.amd64/tmp/legacy/bin/grep+0x473052)
#1 0x4c9cf3 in procline /local/scratch/alr48/cheri/freebsd/usr.bin/grep/util.c:539:8
#2 0x4c8687 in procfile /local/scratch/alr48/cheri/freebsd/usr.bin/grep/util.c:379:18
#3 0x4c6596 in main /local/scratch/alr48/cheri/freebsd/usr.bin/grep/grep.c:714:8

0x62d000008400 is located 0 bytes to the right of 32768-byte region [0x62d000000400,0x62d000008400)
allocated by thread T0 here:
#0 0x493d5d in malloc (/local/scratch/alr48/cheri/build/freebsd-amd64-build/local/scratch/alr48/cheri/freebsd/amd64.amd64/tmp/legacy/bin/grep+0x493d5d)
#1 0x4cad75 in grep_malloc /local/scratch/alr48/cheri/freebsd/usr.bin/grep/util.c:656:13
#2 0x4c8129 in procfile /local/scratch/alr48/cheri/freebsd/usr.bin/grep/util.c
#3 0x4c6596 in main /local/scratch/alr48/cheri/freebsd/usr.bin/grep/grep.c:714:8

SUMMARY: AddressSanitizer: heap-buffer-overflow (/local/scratch/alr48/cheri/build/freebsd-amd64-build/local/scratch/alr48/cheri/freebsd/amd64.amd64/tmp/legacy/bin/grep+0x473052) in regexec

Reviewed By: kevans
MFC after: 1 week

show more ...

grep: replace the internal queue with a ring buffer

We know up front how many items we can have in the queue (-B/Bflag), so
pay the cost of those particular allocations early on.

The reduced queue

grep: replace the internal queue with a ring buffer

We know up front how many items we can have in the queue (-B/Bflag), so
pay the cost of those particular allocations early on.

The reduced queue maintenance overhead seemed to yield about an ~8%
improvement for my earlier `grep -C8 -r closefrom .` test.

MFC after: 2 weeks

show more ...

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

show more ...

Merge ^/head r318560 through r318657.

bsdgrep: Correct per-line line metadata printing

Metadata printing with -b, -H, or -n flags suffered from a few flaws:

1) -b/offset printing was broken when used in conjunction with -o

2) With -o,

bsdgrep: Correct per-line line metadata printing

Metadata printing with -b, -H, or -n flags suffered from a few flaws:

1) -b/offset printing was broken when used in conjunction with -o

2) With -o, bsdgrep did not print metadata for every match/line, just
the first match of a line

3) There were no tests for this

Address these issues by outputting this data per-match if the -o flag is
specified, and prior to outputting any matches if -o but not --color,
since --color alone will not generate a new line of output for every
iteration over the matches.

To correct -b output, fudge the line offset as we're printing matches.

While here, make sure we're using grep_printline in -A context. Context
printing should *never* look at the parsing context, just the line.

The tests included do not pass with gnugrep in base due to it exhibiting
similar quirky behavior that bsdgrep previously exhibited.

Submitted by: Kyle Evans <kevans91@ksu.edu>
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D10580

show more ...

Merge ^/head r317971 through r318379.

bsdgrep: don't allow negative -A / -B / -C

Previously, when given a negative -A/-B/-C argument bsdgrep would
overflow the respective context flag(s) and exhibited surprising
behavior.

Fix this by r

bsdgrep: don't allow negative -A / -B / -C

Previously, when given a negative -A/-B/-C argument bsdgrep would
overflow the respective context flag(s) and exhibited surprising
behavior.

Fix this by removing unsignedness of Aflag/Bflag and erroring out if
we're given a value < 0. Also adjust the type used to track 'tail'
context in procfile() so that it accurately reflects the Aflag value
rather than overflowing and losing trailing context.

This also fixes an inconsistency previously existing between -n and
-C "n" behavior. They are now both limited to LLONG_MAX, to be
consistent.

Add some test cases to make sure grep errors out properly for both
negative context values as well as non-numeric context values rather
than giving bogus matches.

Submitted by: Kyle Evans <kevans91@ksu.edu>
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D10675

show more ...

Merge ^/head r317503 through r317807.

bsdgrep: fix -w flag matching with an empty pattern

-w flag matching with an empty pattern was generally 'broken', allowing
matches to occur on any line whether or not it actually matches -w
criteri

bsdgrep: fix -w flag matching with an empty pattern

-w flag matching with an empty pattern was generally 'broken', allowing
matches to occur on any line whether or not it actually matches -w
criteria.

This fix required a good amount of refactoring to address. procline()
is altered to *only* process the line and return whether it was a match
or not, necessary to be able to short-circuit the whole function in case
of this matchall flag. -m flag handling is moved out as well because it
suffers from the same fate as context handling if we bypass any actual
pattern matching.

The matching context (matches, mostly) didn't previously exist outside
of procline(), so we go ahead and create context object for file
processing bits to pass around. grep_printline() was created due to
this, for the scenarios where the matches don't actually matter and we
just want to print a line or two, a la flushing the context queue and
no -o or --color specified.

Damage from this broken behavior would have been mitigated by the fact
that it is unlikely users would invoke grep -w with an empty pattern.

This was identified while checking PR 105221 for problems it this may
cause in BSD grep, but PR 105221 is *not* a report of this behavior.

Submitted by: Kyle Evans <kevans91 at ksu.edu>
Differential Revision: https://reviews.freebsd.org/D10433

show more ...

IFC @r269962

Submitted by: Anish Gupta (akgupt3@gmail.com)

Merge head from 7/28

Sync to HEAD@r269943.

grep: fix some memory leaks.
Add memory leak fix missing from r268799.

Obtained from: NetBSD

grep: fix some memory leaks.

Bring a couple of changes from NetBSD:

queue.c (CVS Rev. 1.4. 1.5)
Fix memory leaks.
NULL does not need a cast.

grep.c (CVS Rev. 1.6)
Use the more portable getline.

O

grep: fix some memory leaks.

Bring a couple of changes from NetBSD:

queue.c (CVS Rev. 1.4. 1.5)
Fix memory leaks.
NULL does not need a cast.

grep.c (CVS Rev. 1.6)
Use the more portable getline.

Obtained from: NetBSD
MFC after: 3 days

show more ...

- Adjust a comment to actual behaviour
- Makefile nit
- Add more CVS/SVN keywords to make it easier to track changes from NetBSD
in case they add further improvements

Approved by: delphij (mentor)

- Adjust a comment to actual behaviour
- Makefile nit
- Add more CVS/SVN keywords to make it easier to track changes from NetBSD
in case they add further improvements

Approved by: delphij (mentor)
Obtained from: The NetBSD Project

show more ...

UTFize my name.

- Revert strlcpy() changes to memcpy() because it's more efficient and
former may be safer but in this case it doesn't add extra
safety [1]
- Fix -w option [2]
- Fix handling of GREP_OPTIONS [3]

- Revert strlcpy() changes to memcpy() because it's more efficient and
former may be safer but in this case it doesn't add extra
safety [1]
- Fix -w option [2]
- Fix handling of GREP_OPTIONS [3]
- Fix --line-buffered
- Make stdin input imply --line-buffered so that tail -f can be piped
to grep [4]
- Imply -h if single file is grepped, this is the GNU behaviour
- Reduce locking overhead to gain some more performance [5]
- Inline some functions to help the compiler better optimize the code
- Use shortcut for empty files [6]

PR: bin/149425 [6]
Prodded by: jilles [1]
Reported by: Alex Kozlov <spam@rm-rf.kiev.ua> [2] [3],
swell.k@gmail.com [2],
poyopoyo@puripuri.plala.or.jp [4]
Submitted by: scf [5],
Shuichi KITAGUCHI <ki@hh.iij4u.or.jp> [6]
Approved by: delphij (mentor)

show more ...

- Use the traditional behaviour for filename and directory name inclusion
and exclusion patterns [1]
- Some improvements on the exiting code, like replacing memcpy with
strlcpy/strcpy

Approved b

- Use the traditional behaviour for filename and directory name inclusion
and exclusion patterns [1]
- Some improvements on the exiting code, like replacing memcpy with
strlcpy/strcpy

Approved by: delphij (mentor)
Pointed out by: bf [1], des [1]

show more ...

Add BSD grep to the base system and make it our default grep.

Deliverables: Small and clean code (1,4 KSLOC vs GNU's 8,5 KSLOC),
lower memory usage than GNU grep, GNU compatibility,

Add BSD grep to the base system and make it our default grep.

Deliverables: Small and clean code (1,4 KSLOC vs GNU's 8,5 KSLOC),
lower memory usage than GNU grep, GNU compatibility,
BSD license.

TODO: Performance is somewhat behind GNU grep but it is only
significant for bigger searches. The reason is complex, the
most important factor is that GNU grep uses lots of
optimizations to improve the speed of the regex library.
First, we need a modern regex library (practically by adopting
TRE), add support for GNU-style non-standard regexes and then
reevalute the performance issues and look for bottlenecks. In
the meantime, for those, who need better performance, it is
possible to build GNU grep by setting WITH_GNU_GREP.

Approved by: delphij (mentor)
Obtained from: OpenBSD (http://www.openbsd.org/cgi-bin/cvsweb/src/usr.bin/grep/),
freegrep (http://github.com/howardjp/freegrep)
Sponsored by: Google SoC 2008
Portbuild tests run by: kris, pav, erwin
Acknowledgements to: fjoe (as SoC 2008 mentor),
everyone who helped in reviewing and testing

show more ...