| Age | Commit message (Collapse) | Author |
|---|
|
|
|
(#3464)
|
|
Testing has uncovered another case where drops can spin trying to lock a
checkpoint handle until a checkpoint completes. This change fixes that
in two ways: attempting to lock (but not open) a handle won't spin, and
drop will always attempt to lock the live tree before locking any
checkpoint handles.
|
|
Added a script to merge workgen and WT outputs. Make workgen_stat.sh script agnostic as to what analyze tool to run.
|
|
Added wtperf.py script to run wtperf files. Added some supporting functions in the runner module to used by programs created by this tool.
|
|
|
|
(#3456)
* WT-3361 Resolve Windows build warnings, build more test programs on Windows.
Update SConstruct with entries for all test programs (not all build
and/or run yet).
Replace all references to pthread_create, pthread_join and pthread_t
with __wt_thread_create, __wt_thread_join and wt_thread_t, that is,
use the WiredTiger versions for Windows portability.
* Don't try and smoke-test bloom as part of the pull builds, it won't
necessarily run.
* Make WT's thread create / join functions visible for ex_thread.
* Build all test programs with shim and wtlibs, it's simpler and allows
for future changes without touching the SConstruct file.
|
|
|
|
* Verbose messages for lookaside activity are generated once-per-checkpoint.
|
|
|
|
|
|
* WT-3356 Use atomic reads of rwlocks.
Previously we had some conditions that checked several fields within a rwlock by indirecting to the live structure. Switch to always doing a read of the full 64-bit value, then using local reads from the copy.
Otherwise, we're relying on the compiler and the memory model to order the structure accesses in "code execution order". That could explain assertion failures and/or incorrect behavior with the new rwlock implementation.
* Change all waits to 10ms.
Previously when stalling waiting to get into the lock we would wait for 1ms, but once queued we waited forever. The former is probably too aggressive (burns too much CPU when we should be able to wait for a notification), and the latter is dangerous if a notification is ever lost (a thread with a ticket may never wake up).
|
|
|
|
* WT-3354 Fix bugs found by Coverity.
* two cases where error checking for rwlocks should goto the error label for cleanup.
* LSM code not restoring isolation if a checkpoint fails part way through
* Take care with ordering an assertion after a read barrier.
We just had an assertion failure on PPC, and from inspection it looks
like read in the assertion could be scheduled before read that sees the
ticket allocated. We have a read barrier in this path to protect
against exactly that kind of thing happening to application data, move
the assertion after it so our diagnostics are also safe.
|
|
* Add a workload that stresses rwlock performance under various conditions (including `threads >> cores`), tune read and write lock operations to only spin when it is likely to help, and to back off to a condition variable when there is heavy contention.
* New rwlock implementation: queue readers and writers separately, don't enforce fairness among readers or if the lock is overwhelmed.
* Switch to a spinlock whenever we need to lock a page.
Previously we had a read/write lock in the __wt_page structure that was only ever acquired in write mode, plus a spinlock in the page->modify structure. Switch to using the spinlock for everything.
One slight downside of this change is that we can no longer precisely determine whether a page is locked based on the status of the spinlock (since another page sharing the same lock could be holding it in the places where we used to check). Since that was only ever used
for diagnostic / debugging purposes, I think the benefit of the change outweighs this issue.
* Fix a bug where a failure during `__wt_curfile_create` caused a data handle to be released twice. This is caught by the sanity checking assertions in the new read/write lock code.
* Split may be holding a page lock when restoring update. Tell the restore code we have the page exclusive and no further locking is required.
* Allocate a spinlock for each modified page.
Using shared page locks for mulitple operations that need to lock a page (including inserts and reconciliation) resulted in self-deadlock when the lookaside table was used. That's because reconciliation held a page lock, then caused inserts to the lookaside table, which acquired the page lock for a page in the lookaside table. With a shared set of page locks, they could both be the same lock.
Switch (back?) to allocating a spinlock per modified page. Earlier in this ticket we saved some space in __wt_page, so growing __wt_page_modify is unlikely to be noticeable.
* Tweak padding and position of the spinlock in WT_PAGE_MODIFY to claw back some bytes.
Move evict_pass_gen to the end of WT_PAGE: on inspection, it should be a cold field relative to the others, which now fit in one x86 cache line.
|
|
* Simplify Windows include files, clean up Windows warnings.
* Remove incorrect path from copyright skip list.
* Clean up a set of places where an operation could potentially truncate a value by performing the operation at the larger size and then casting the result.
|
|
failure (#3445)
* WT-3307 Close btree/dhandle when error handling __wt_checkpoint_close failure
* Let EBUSY be returned immediately without further processing dhandles.
* Minor KNF
|
|
Only give up on update/restore if we don't use any updates.
Previously, we were tracking the size of saved updates vs the size of
uncommitted updates: renamed the variables for clarity. Only give up on
update/restore eviction if (1) the page doesn't split and (2) all
updates are saved -- i.e., we're not applying anything.
|
|
|
|
|
|
Pulling in the 2.9.2 release notes
|
|
|
|
held. (#3443)
Closing the file handles that remain open at database close will attempt
to unlock a lock that's not currently held.
|
|
|
|
Python. (#3432)
|
|
|
|
handle (#3436)
This is a problem for LSM or any non-standard data source that calls
CURSOR_UPDATE_API_CALL with an in-memory configuration and a NULL btree
handle.
Split CURSOR_UPDATE_API_CALL into two versions, one of which expects a
btree handle, one of which doesn't.
Rename the argument to TXN_API_CALL_NOCONF from "bt" to "dh", it's a
data handle in that macro, not a btree handle.
|
|
(#3406)" (#3434)
This reverts commit a3073be71b1406c7eef60e35fcf2cdc1174e752b.
|
|
Document and implement a WT_CURSOR.modify API
|
|
|
|
time (#3430)
|
|
* WT-3264 Fix log file cleanup when toggling logging.
* whitespace
* Fix test loop bug
* Remove debugging. Test fixes. Reset slot.
* Update comments. Remove unused var.
* Rename goto label
|
|
|
|
Don't acquire the eviction walk-lock when releasing exclusive eviction access to a file, it can deadlock. Instead, atomically decrement the counter.
|
|
|
|
A correctly-written WiredTiger application will have an error handler
for WT_PANIC, and that isn't explicit in the documentation.
|
|
|
|
Fix up some copyright notices.
|
|
|
|
Use awk instead of wc to get a count of lines, awk never includes
whitespace in the output.
|
|
We use a pragma on Windows to force a struct to be packed, but were
missing the "end" pragma that restores normal layout. The result was
that most structs were being packed, leading to poor performance for
workloads (particularly when accessing session structures).
|
|
Use awk instead of wc to get a count of lines, awk never includes
whitespace in the output.
|
|
We use a pragma on Windows to force a struct to be packed, but were
missing the "end" pragma that restores normal layout. The result was
that most structs were being packed, leading to poor performance for
workloads (particularly when accessing session structures).
|
|
* WT-3312 encryption btree configuration test
Don't check for the metadata file explicitly (it implies we should check
other special files too, like the LAS file). If there's no explicit
encryption argument, default to the connection's value, that includes
the metadata file and the LAS file.
* Remove my added line in the comment, it didn't add anything useful.
|
|
This reverts commit 48c7cf59ccf583369fa98733b388983bd4abb70e.
|
|
The workload generator runs core workloads in C++, which directly calls the C API and thus minimizes the overhead of the test program. The workload setup is done in Python to allow for an expressive workload definition language.
Initial commit has a number of example workloads that match equivalent wtperf workloads.
|
|
* Changes for SERVER-28166: Assess effects of pinning a lot of content in storage engine cache
Instead of counting the update chains we skip/don't-skip, calculate the memory held in the update chain. Then if there are skipped updates in the chain, count that memory as memory we won't get back if we rewrite the page, else count it as memory we will recover. Change the test for rewriting a page in memory from a percentage of the update chains that we skipped/didn't-skip to whether or not we'll recover at least 2KB from rewriting the page. This change should avoid rewriting pages in memory where we're just wasting time, that is, tiny pages or pages without any updates on them.
Reconciliation has a test in __rec_write_check_complete() of whether an update-restore page reconciliation should fail because there aren't enough updates being discarded that it's worth rewriting the page in memory. That test incorrectly used the count of currently allocated boundary structures in its calculations, not the count of boundary structures used in a specific reconciliation. That bug led to eviction repeatedly doing update-restore reconciliation and rewriting pages in memory, to little or no gain, instead of switching to the LAS table.
Further, the test was intended to be if 10% of the update chains won't have to be re-instantiated when the page is rewritten. I think the test was wrong, but regardless, implementing the test correctly doesn't help with the test load I'm using. (My test is creating a snapshot and then inserting records until the cache is full.) Re-instantiating the page in memory stalls in that test case, which makes sense because rewriting a page in memory isn't evicting anything, it's just shrinking the memory being pinned down.
Change the test: if we can discard a chunk of the original page, rewrite the page in memory. Else, fallback to the lookaside table if we think lookaside will work.
* Try to shorten update chains during eviction.
* The memory in the update chain should include memory held by aborted transactions. Rather than put another test inside the loop, split the loop into two parts, eviction and checkpoint. Checkpoint doesn't need to calculate the memory in the update chain, nor does it have to track the minimum transaction ID.
* Add a smoke test for the LAS table.
* Skip reserved items when inserting update records into the LAS table, they're never restored.
* Don't fall back to the LAS table when configured for in-memory eviction.
* Replace __wt_evict's test of WT_EVICT_IN_MEMORY in with a test of WT_CONN_IN_MEMORY, there's no reason to further complicate the pass-back of the flags value to that function.
* Ignore lookaside table collision tests and checks to avoid rewriting pages in memory for little gain, when configured in-memory. The former isn't an issue for in-memory configurations, and small caches with in-memory configurations can force us to rewrite every possible page.
|
|
|
|
successfully (#3405)
* Introduce a thread stuct with a member denoting if thread got created
* Change windows thread create in accordance to new thread struct
* Fix build error
|
|
|
