1From wiggans@aipl.arsusda.gov Mon Sep 12 11:05:58 1994 2Received: from vangogh.CS.Berkeley.EDU by python.bostic.com (8.6.9.Beta4/2.6) 3 id OAA16853; Mon, 12 Sep 1994 14:05:42 -0400 4From: wiggans@aipl.arsusda.gov 5Received: from hofmann.CS.Berkeley.EDU (hofmann.CS.Berkeley.EDU [128.32.34.35]) by vangogh.CS.Berkeley.EDU (8.7.Alpha.1/8.6.9.Beta0) with ESMTP id LAA15825 for <bostic@vangogh.CS.Berkeley.EDU>; Mon, 12 Sep 1994 11:05:20 -0700 (PDT) 6Received: from uu7.psi.com (uu7.psi.com [38.145.204.6]) by hofmann.CS.Berkeley.EDU (8.6.9/8.6.6.Beta11) with SMTP id LAA25681 for <bostic@cs.berkeley.edu>; Mon, 12 Sep 1994 11:05:44 -0700 7Received: from AIPL.ARSUSDA.GOV by uu7.psi.com (5.65b/4.0.071791-PSI/PSINet) via SMTP; 8 id AA00699 for bostic@cs.berkeley.edu; Mon, 12 Sep 94 14:06:15 -0400 9Received: by aipl.arsusda.gov (AIX 3.2/UCB 5.64/4.03) 10 id AA14802; Mon, 12 Sep 1994 14:05:48 -0400 11Message-Id: <9409121805.AA14802@aipl.arsusda.gov> 12Subject: db 1.85 problem 13To: bostic@cs.berkeley.edu (Keith Bostic) 14Date: Mon, 12 Sep 1994 14:05:47 -0400 (EDT) 15X-Mailer: ELM [version 2.4 PL22] 16Content-Type: text 17Content-Length: 2553 18Status: RO 19 20In using the btree option to sequentially read and then write a file, we 21are having a problem with 1.85. When compiled with 1.73 there is no 22problem. The problem is that the seq call keeps reading the same record. 23The code follows: 24 25/* chkseq.c Check sequential read and write */ 26 27#include <stdio.h> 28#include <sys/stat.h> 29#include <string.h> 30#include <stdlib.h> 31#include <fcntl.h> /* O_CREAT, O_RDWR */ 32#include <errno.h> /* Error numbers */ 33#include <db.h> 34 35extern int errno; 36extern char *sys_errlist[]; 37 38typedef struct idst { 39 char id1[7]; 40} id; 41 42void cvtid(char *, char *); 43 44void main() { 45 char anim10[11], datastor[212],keystor[10], *pc; 46 int i; 47 long in = 0L; 48 DB *dbp, *dbpo; 49 DBT key, data, keyo, datao; 50 51 if ((dbp = dbopen("bullxrf.db", O_RDWR, 0664 52 , DB_BTREE, NULL )) == NULL) { 53 printf("\n Error on dbopen %d %s\n",errno,strerror(errno)); 54 exit(61); 55 } 56 key.size = 7; 57 keyo.size = 7; 58 59 while (dbp->seq(dbp, &key, &data,R_NEXT) == 0) { 60 in++; 61 if (in > 20) break; 62/* pc = (char *) key.data; 63for (i=0;i<key.size;i++) printf("%02x",pc[i]); printf("\n"); */ 64 cvtid(anim10,key.data); printf("%s\n",anim10); 65 memcpy(keystor,key.data,key.size); 66/* for (i=0;i<key.size;i++) printf("%02x",keystor[i]); printf("\n"); */ 67 memcpy(datastor,data.data,data.size); 68/* for (i=0;i<8;i++) printf("%02x",datastor[i]); printf("\n"); */ 69 keyo.data = keystor; 70 datao.data = datastor; 71 datao.size = data.size; 72/* 73 if (in % 1000 == 1) { 74 cvtid(anim10,key.data); printf("%5d %s\n",in,anim10); */ 75 if (dbp->put(dbp, &keyo, &datao,0) != 0) { 76 printf("Write failed at %d\n",in); 77 exit(85); 78 } 79/* } 80 */ 81 } 82 printf("%d Records copied\n",in); 83 dbp->close(dbp); 84} 85 86I am running on an RS/6000 AIX 3.2.5. The section of the make file 87follows: 88 89# Make file 90all: chkseq 91 92chkseq: chkseq.c 93 cc -gO3 -lm -o chkseq\ 94 -L /data6/hash/include/sys/lib -l db -I /data6/hash/include \ 95 chkseq.c cvtid.o ascii.o 96# -L /data12/db.1.85 -l db -I /data12/db.1.85/include \ 97 98We would appreciate your help. 99Thanks, 100 101-- 102George Wiggans I================================================I 103 |Animal Improvement Programs Laboratory | 104Phone: 301-504-8407 |Bldg 263 Beltsville Agricultural Research Center| 105FAX: 301-504-8092 |Beltsville, MD 20705-2350 USA | 106wiggans@aipl.arsusda.gov | | 107=========================I================================================I 108 109From wiggans@aipl.arsusda.gov Fri Sep 16 20:27:22 1994 110Received: from vangogh.CS.Berkeley.EDU by python.bostic.com (8.6.9.Beta4/2.6) 111 id XAA09260; Fri, 16 Sep 1994 23:27:09 -0400 112From: wiggans@aipl.arsusda.gov 113Received: from hofmann.CS.Berkeley.EDU (hofmann.CS.Berkeley.EDU [128.32.34.35]) by vangogh.CS.Berkeley.EDU (8.7.Alpha.1/8.6.9.Beta0) with ESMTP id UAA25674 for <bostic@vangogh.CS.Berkeley.EDU>; Fri, 16 Sep 1994 20:27:03 -0700 (PDT) 114Received: from uu7.psi.com (uu7.psi.com [38.145.204.6]) by hofmann.CS.Berkeley.EDU (8.6.9/8.6.6.Beta11) with SMTP id UAA15043 for <bostic@cs.berkeley.edu>; Fri, 16 Sep 1994 20:27:16 -0700 115Received: from AIPL.ARSUSDA.GOV by uu7.psi.com (5.65b/4.0.071791-PSI/PSINet) via SMTP; 116 id AA18737 for bostic@cs.berkeley.edu; Fri, 16 Sep 94 23:27:14 -0400 117Received: by aipl.arsusda.gov (AIX 3.2/UCB 5.64/4.03) 118 id AA10907; Fri, 16 Sep 1994 23:26:18 -0400 119Message-Id: <9409170326.AA10907@aipl.arsusda.gov> 120Subject: Test case 121To: bostic@cs.berkeley.edu (Keith Bostic) 122Date: Fri, 16 Sep 1994 23:26:16 -0400 (EDT) 123X-Mailer: ELM [version 2.4 PL22] 124Content-Type: text 125Content-Length: 3713 126Status: RO 127 128The following program loads 2 10 character animal ID which are used to 129change an animal's ID. After loading, it closes, then opens and 130sequentially reads and rewrites the file changing the first character of 131the 2nd ID to U. Failure is observed when the update part gets stuck on 132the first record, rereading it. The last step displays the updated file. 133The name of the data file is a command line argument. 134 135The data: 136A000027875A000135891 137A000059165A000130168 138A000060256A000133490 139A040025906A000136770 140A040027881A000135829 141A040028611A000137873 142A040032413A000056974 143A040050163A000126233 144A040050329A000126177 145A040050411A000119017 146A040050995A000116767 147A040051022A000126669 148A040051276A000127444 149A040051514A000120563 150A040051597A000127287 151A040051627A000127284 152A040051700A000126914 153A040051810A000127286 154A040051964A000118834 155A040052164A000135104 156A040052165A000127688 157A040052186A000126926 158A040052530A000126287 159A040052560A000119160 160A040052892A000125334 161A040053004A000127684 162A040053359A000128628 163A040053378A000137680 164A040053416A000128825 165A040053589A000120369 166A040053620A000128460 167A040053751A000123525 168A040053754A000126736 169A040054191A000126286 170A040054251A000121745 171A040054253A000127848 172A040054596A000130931 173A040054981A000128731 174A040055000A000127689 175 176The program: 177/* chkseq.c Check sequential read and write */ 178 179#include <stdio.h> 180#include <sys/stat.h> 181#include <string.h> 182#include <stdlib.h> 183#include <fcntl.h> /* O_CREAT, O_RDWR */ 184#include <errno.h> /* Error numbers */ 185#include <db.h> 186 187extern int errno; 188extern char *sys_errlist[]; 189 190 191void main(int argc, char *argv[]) { 192 char id1[] = {" "}, id2[] = {" "}; 193 int i; 194 long in = 0L, out = 0L; 195 DB *dbp, *dbpo; 196 DBT key, data, keyo, datao; 197 FILE *fopen(), *fin; 198 199 if ((fin = fopen(argv[1],"r")) == NULL) { 200 printf("Unable to open %s\n",argv[1]); 201 exit(25); 202 } 203 if ((dbp = dbopen("test.db",O_RDWR | O_CREAT, 0664 204 , DB_BTREE, NULL )) == NULL) { 205 printf("\n Open error on test.db %d %s\n",errno,strerror(errno)); 206 exit(25); 207 } 208 209 while (fscanf(fin," %10s%10s",id1,id2) > 0) { 210 key.size = 11; 211 data.size = 11; 212 key.data = id1; 213 data.data = id2; 214 printf("%10s %10s\n",key.data,data.data); 215 if (dbp->put(dbp, &key, &data,R_NOOVERWRITE) != 0) { 216 printf("Error writing output\n"); 217 } 218 out++; 219 } 220 printf("%d Records in\n",out); 221 dbp->close(dbp); 222 223 if ((dbp = dbopen("test.db", O_RDWR, 0664 224 , DB_BTREE, NULL )) == NULL) { 225 printf("\n Error on dbopen %d %s\n",errno,strerror(errno)); 226 exit(61); 227 } 228 229 while (dbp->seq(dbp, &key, &data,R_NEXT) == 0) { 230 strcpy(id1,key.data); 231 keyo.size = 11; 232 datao.size = 11; 233 keyo.data = id1; 234 strcpy(id2,data.data); 235 id2[0] = 'U'; 236 datao.data=id2; 237 printf("%10s %10s\n",key.data,data.data); 238 in++; 239 if (in > 50) break; 240 if (dbp->put(dbp, &keyo, &datao,0) != 0) { 241 printf("Write failed at %d\n",in); 242 exit(85); 243 } 244 } 245 printf("%d Records copied\n",in); 246 in = 0; 247 dbp->seq(dbp, &key, &data,R_FIRST); 248 printf("%10s %10s\n",key.data,data.data); 249 in++; 250 while (dbp->seq(dbp, &key, &data,R_NEXT) == 0) { 251 in++; 252 printf("%10s %10s\n",key.data,data.data); 253 } 254 printf("%d Records read\n",in); 255 dbp->close(dbp); 256} 257 258 259-- 260George Wiggans I================================================I 261 |Animal Improvement Programs Laboratory | 262Phone: 301-504-8407 |Bldg 263 Beltsville Agricultural Research Center| 263FAX: 301-504-8092 |Beltsville, MD 20705-2350 USA | 264wiggans@aipl.arsusda.gov | | 265=========================I================================================I 266 267From bostic Fri Sep 23 08:44:56 1994 268To: wiggans@aipl.arsusda.gov /usr/src/local/db/test/SEQ_TEST/mbox 269Subject: Re: Test case 270 271 272OK, I've attached a tentative patch for the bug, that appears 273to fix it on my local system. Please let me know if you have 274any further problems with this. 275 276There are a couple of issues here. The first, is that to some 277extent, the btree code is correct. You aren't replacing the 278cursor in your test program, you're adding a new key, which 279just happens to be where the cursor was. So, the btree code 280is doing you a favor by returning the new key as part of the 281cursor walk, and it's not its fault. ;-} 282 283However, because a put to the cursor record is done using a 284delete/add pair, doing it the "right" way will result in the 285exact same behavior as you saw doing it "wrong". 286 287Thinking about this further, there's another bug that's going to 288hit eventually -- if you have duplicate records, the current 289scheme of doing delete/add to replace the cursor record can result 290in a record being returned twice, which is tacky at best, if not 291actually wrong. 292 293I think I may have to revisit how duplicate records are stored. 294Which does not make me happy. ;-{ 295 296--keith 297 298*** db/btree/bt_seq.c.orig Fri Sep 23 08:35:06 1994 299--- db/btree/bt_seq.c Fri Sep 23 08:34:58 1994 300*************** 301*** 35,41 **** 302 */ 303 304 #if defined(LIBC_SCCS) && !defined(lint) 305! static char sccsid[] = "@(#)bt_seq.c 8.7 (Berkeley) 7/20/94"; 306 #endif /* LIBC_SCCS and not lint */ 307 308 #include <sys/types.h> 309--- 35,41 ---- 310 */ 311 312 #if defined(LIBC_SCCS) && !defined(lint) 313! static char sccsid[] = "@(#)bt_seq.c 8.8 (Berkeley) 9/23/94"; 314 #endif /* LIBC_SCCS and not lint */ 315 316 #include <sys/types.h> 317*************** 318*** 246,252 **** 319 PAGE *h; 320 indx_t index; 321 pgno_t pg; 322! int exact; 323 324 /* 325 * There are a couple of states that we can be in. The cursor has 326--- 246,252 ---- 327 PAGE *h; 328 indx_t index; 329 pgno_t pg; 330! int exact, rval; 331 332 /* 333 * There are a couple of states that we can be in. The cursor has 334*************** 335*** 255,269 **** 336 c = &t->bt_cursor; 337 338 /* 339! * The cursor was deleted where there weren't any duplicate records, 340! * so the key was saved. Find out where that key would go in the 341! * current tree. It doesn't matter if the returned key is an exact 342! * match or not -- if it's an exact match, the record was added after 343! * the delete so we can just return it. If not, as long as there's 344! * a record there, return it. 345 */ 346! if (F_ISSET(c, CURS_ACQUIRE)) 347! return (__bt_first(t, &c->key, ep, &exact)); 348 349 /* Get the page referenced by the cursor. */ 350 if ((h = mpool_get(t->bt_mp, c->pg.pgno, 0)) == NULL) 351--- 255,299 ---- 352 c = &t->bt_cursor; 353 354 /* 355! * The cursor was deleted and there weren't any duplicate records, 356! * so the cursor's key was saved. Find out where that key would 357! * be in the current tree. If the returned key is an exact match, 358! * it means that a key/data pair was inserted into the tree after 359! * the delete. We could reasonably return the key, but the problem 360! * is that this is the access pattern we'll see if the user is 361! * doing seq(..., R_NEXT)/put(..., 0) pairs, i.e. the put deletes 362! * the cursor record and then replaces it, so the cursor was saved, 363! * and we'll simply return the same "new" record until the user 364! * notices and doesn't do a put() of it. Since the key is an exact 365! * match, we could as easily put the new record before the cursor, 366! * and we've made no guarantee to return it. So, move forward or 367! * back a record if it's an exact match. 368! * 369! * XXX 370! * In the current implementation, put's to the cursor are done with 371! * delete/add pairs. This has two consequences. First, it means 372! * that seq(..., R_NEXT)/put(..., R_CURSOR) pairs are going to exhibit 373! * the same behavior as above. Second, you can return the same key 374! * twice if you have duplicate records. The scenario is that the 375! * cursor record is deleted, moving the cursor forward or backward 376! * to a duplicate. The add then inserts the new record at a location 377! * ahead of the cursor because duplicates aren't sorted in any way, 378! * and the new record is later returned. This has to be fixed at some 379! * point. 380 */ 381! if (F_ISSET(c, CURS_ACQUIRE)) { 382! if (rval = __bt_first(t, &c->key, ep, &exact)) 383! return (RET_ERROR); 384! if (!exact) 385! return (rval); 386! /* 387! * XXX 388! * Kluge -- get, release, get the page. 389! */ 390! c->pg.pgno = ep->page->pgno; 391! c->pg.index = ep->index; 392! mpool_put(t->bt_mp, ep->page, 0); 393! } 394 395 /* Get the page referenced by the cursor. */ 396 if ((h = mpool_get(t->bt_mp, c->pg.pgno, 0)) == NULL) 397 398 399 400